advancing testing program maturity in your organization
TRANSCRIPT
Intended for Knowledge Sharing only
Advancing Testing Program MaturityOptimizely User Group – San Francisco
Oct 2017
Intended for Knowledge Sharing only
RAMKUMAR RAVICHANDRAN
Intended for Knowledge Sharing only2
Director, Analytics at Visa, Inc.
Data Driven Decision Making via
Insights from Analytics, A/B
Testing and Research
Manager, Analytics & AB Testing
Data Driven Decision Making via
Insights from Analytics, A/B
Testing and Research
ROGER CHANG MIMI LE
Senior Director
Product Launch Management
Intended for Knowledge Sharing only
Quick recap of what it is
Intended for Knowledge Sharing only
Just venting it out a little…
3
JUST CHANGE THE COLOR OF THE BUTTON, DAMN IT!
Intended for Knowledge Sharing only
WAIT A MINUTE! WHAT WILL YOU DO WITH THAT MUCH MONEY, SENORE?
Intended for Knowledge Sharing only
YOU BROKE IT, DUDE!!!
Intended for Knowledge Sharing only
THE NUMBERS AREN’T IN LINE – YOU MUST HAVE SCREWED UP!
Intended for Knowledge Sharing only
WHY WOULD I NEED YOU, WHEN I GOT ARTIFICIAL INTELLIGENCE?
Intended for Knowledge Sharing only
Intended for Knowledge Sharing only
Quick recap of what it is
Intended for Knowledge Sharing only
A typical Testing Program in Utopia…
9
IF GOD DECIDE TO CREATE AN A/B TEST PROGRAM, WHAT WOULD IT LOOK LIKE…
Intended for Knowledge Sharing only
Every major product change
has been iterated, quantified
& contextualized
A centralized but modular,
seamless & integrated Learn,
Listen and Test Framework
covering all domains
A Single-Source-Of-Truth
Testing Datamart within the
Organization’s Datalake for
year end Program
effectiveness studies
Unified Workflow & Project
Management with searchable
Knowledge repository &
centralized Admin capabilities
Programmatic Testing with
human intervention protocols
Intended for Knowledge Sharing only
Quick recap of what it is
Intended for Knowledge Sharing only
What is a Testing Maturity Curve?
11
TESTING PROGRAM MATURITY CURVE
SELL
SCALE
EXPAND
DEEPEN
TRANSFORM
Phases of Maturity
Va
lue
Ad
d
We are here
DEEPEN
• Content Delivery, Personalization,
Champion-Challenger Set up
• Platform: Cross Integration with
Analytics, ML, Session Replays &
Research at Application Layer
• Predictive Analytics on Test Impact:
Test-Mix Models (Scenario Planning
& Scoring)
• Unified Workflow & Project
Management
EXPAND
• Complexity and scope of the tests
• Multi (Variate, Pages, Experience,
Device, Domain)
• Enterprise Framework (Server Side
Integration, Datamart)
• Enterprise rollup of Operational and
Strategic Impact
• Searchable Knowledge bank &
Feedback loop into Design Stage
TRANSFORM
TESTING PROGRAM MATURITY- PHASES
PHASES KEY ACTION ITEMS SUCCESS CRITERIA
SELL
• Buy-ins across leadership &
stakeholders
• Scrappy quick win tests
• Allaying fears of Dev/QA/Security org
• Tangible KPI impact
• Sponsor Business Units and victory
use cases: Prod, UX, Mktg
• Approval for a cross functional team
and Testing environment set up
SCALE
• Agile Workflow set up
• Test Pipeline created & shared
• Testing Dashboard
• Readouts shared with stakeholders
• A successful rollout because of Test
& Learn Initiative (Use Case Driven-
>Numbers Driven->Experience
Driven)
• Testing formalized within Dev Cycle
• Algorithmic Test Management (Traffic
adjustments, winner ramps,
combinatorial tests)
• Test Modularity & Portability
• Testing as “Monetizable” Product
• Test & Learn made self serve via
Trainings for Citizen Experimenters
• Cross Pollination across BU – within
the DNA of the organization
TRANSFORM
TESTING PROGRAM MATURITY- EXPERIENCE & LEARNING
PHASES CHALLENGES RESOLUTION THAT WORKED FOR US
SELL• Executive Buy-ins
• Pushback from Security, Branding,
Integration, Development & QA teams
• Proof-Of-Concept (guard against
weak POC & Sponsor BU)
• Risk Ownership with executive air
cover/Shared limelight
• CMS/Bug Fixes – Good and bad!
SCALE
• Resourcing & Funding support:
Availability & size of shared team
• Sandbox availability/sync with
release cycles/broken tagging
• Production ramp
• Show progress even with persisting
challenges
• Successful Project delivery
• Dashboards & Communication
readouts
EXPAND
• Sample size, cookie issues and cross
domain traffic. Interaction problems.
• Consistency and integration issues of
tagging and logic between front and
back end and within backends itself
• Knowledge Management site and
dashboard
• Instrumentation request for the
Engineering team to link the various
cookies and identifiers
DEEPEN
• Huge investment and potential
tradeoffs with re-architecting
instrumentation
• Resourcing and Funding into platform
set-up & product builds
• Potentially Test-Mix Models on
manually scraped metadata (less
rigorous)
• Server Side product set up
• Dependent on successful transition
from previous phase
• Resourcing & funding
• Test & Learn made self serve via
Trainings for Citizen Experimenters.
Brown bags, whitepapers?
Intended for Knowledge Sharing only
Quick recap of what it is
Intended for Knowledge Sharing only
What has worked for us so far…
15
KNOWING WHAT WE ARE TESTING & HOW MUCH TO EXPECT…
Message
Prominence
Flow
Form
Clear and crisp Value Prop and Call to Action (CTA)
Trendy and easy to spot
Easily spotted and fitting with the Consumer’s mental model
Navigation and Pathing
Minimal and relevant elements only
Placement
Personalization Personalized with behavioral insights
Content Algorithmic delivery/Contextualized Content
Performance Platform Performance (Latency, Uptime, Errors)
Exp
ecte
d Im
pac
t
Intended for Knowledge Sharing only
PLANNING IT END-TO-END
• Analytics team creates direct/proxy metrics to measure the performance
• Instrument metrics if needed
• Decision on the Research Methodology based on Analytical findings
AC
TIO
NS
• Defined the question to be answered and why, Design the changes, know the cost and finalize success criteria
• Quantify/Analyze the impact
• Size the potential impact on launching
Measure LaunchStrategy
PH
ASES
Analyze
Primary Metrics, e g.,
• Click Through Rate• NPS
Secondary Metrics
• Repeat Visits• Lifetime Value
Questions
• Target Customers• Where and What is
being checked?• Why is this even
being considered?• Target Metrics and
success criteria
Research Methods
• Attitudinal vs. Behavioral
• Qualitative vs. Quantitative
• Context for Product Use
Factors deciding Research Methods
• Speed of execution• Cost of execution• Reliability• Product
Development Stage
Factors deciding eventual rollout (in order of priority)
• Strategic need• Estimated impact
calculation from Analytics
• Findings from other sources (Data Analytics/Mining, Consumer Feedback
DET
AIL
S
Intended for Knowledge Sharing only
KNOWING WHEN TO SET UP A/B TEST AND NOT…
Method DescriptionFactors
Speed Cost Inference Dev Stage
Prototyping
Usability
Studies
Focus Group
Surveys &
Feedback
Pre-Post
A/B Testing
Create & Test prototypes
internally (external, if
needed)
Standardized Lab
experiments – Panel/s of
employees/friends/family
In-depth interviews for
Feedback
Email/Pop-ups Surveys
Roll-out the changes and
then test for impact
Different experiences to
users and then measure delta
Quickest (HTML
Prototypes)
Quick (Panel,
Questions, Read)
Slow (+Detailed
interviews)
Slower
(+Response rate)
Slower (Dev+QA+
Launch+Release
cycle)
Slowest
(+Sampling+
Profiling+
Statistical
Inferencing)
Inexpensive
(Feedback
incentives)
Relatively
expensive
(+Lab)
Expensive
(+Incentive
+Time)
Expensive
(Infra to
send, track
& Read)
Costly
(+Tech
resources)
Very Costly
(+Tech
+Analytics
+Time)
Directional
+Consistency across
users
+additional context
on Why?
+strength of
numbers
+Possible Statistical
Significance but risk
of bad experience.
+Rigorous (Statistical
Significance). *Risk of
bad experience
reduced.
Ideation Stage
Ideation Stage
Ideation Stage
Ideation/Dev/
Post Launch
Post Launch
Pre Launch
(after Dev)
Intended for Knowledge Sharing only
PAYING YOUR DUES –RIGHT WORKFLOW MANAGEMENT & XFUNCTIONAL OWNERSHIP
• A/B Test Analyst (Analytics): The driver of the testing program. Involved from start to finish up until the hand-off of a successful test to its respective product owner. A SME in the Optimizely tool, owner of test setup, deployment, and analysis.
• Product Partner: Talks to and brings in the right people for different steps of the process. Offers product’s perspective in terms of gatekeeping duties on test ideas. Well connected to different product owners and acts as the liaison towards the product team.
• QA Partner: Helps ensure that there are no bugs in the test setup, from a usability standpoint.
• Technology Partner: Offers consultation on feasibility for tests, assists in setup of advanced tests.
• Design Partner: Helps the team germinate ideas, as well as give the team visuals to work off of in a test.
IdeationPrioritization /
GroomingSetup QA Deployment Analysis Implementation
Analytics, Product, Design, Tech
Analytics, QA
Analytics, Product
Intended for Knowledge Sharing only
…AND FOCUS ON FULL SPECTRUM OF OPERATIONAL METRICS
Operational Program KPIs
• # of Tests run per month
• % Successful tests
• % Learning Tests
• % Workaround/Bug fix Tests
• #Channels Tested on
• Time from ideation to deployment
• Time from test outcome to product implementation
• Program RoI
• Stakeholder NPS
• KPI Delta vs. Universal Control
Intended for Knowledge Sharing only
…both raw and
YoY growth
forms
& ANALYTICS VALUE CHAIN: STRATEGY DRIVES EVERY INITIATIVE & ANALYTICS
MEASURES ITS EFFECTIVENESS!
Analytics provides insights into “actions”, Research context on “motivations” & Testing
helps verify the “tactics” in the field and everything has to be productized…
Strategy
Data Tagging
Data Platform
Reporting
Analytics
Research
Cognitive
IterativeLoop Key benefits
Focus on Big Wins
Reduced Wastage
Quick Fixes
Adaptability
Assured execution
Learning for future
initiatives
Intended for Knowledge Sharing only
Optimization
…& TIMING IT CORRECTLY WITHIN THE ANALYTICS MATURITY RAMP
Testing makes sense after we know what the baseline actually looks like…
Intended for Knowledge Sharing only
60%
20%
10%5% 5%
20%
30%
15%
10%5%
20%
25%
25%
25%
20%
25%
25%
20%
15%
25%
20%
20%
20%
20%
15%
YEAR 1 YEAR 2 YEAR 3 YEAR 4 YEAR 5
Primary source of insights for Decision Making along the Analytics Maturity Curve
Reporting Data Analytics User Research A/B Testing Advanced Analytics/Machine Learning Data Products Cognitive Analytics
ILLUSTRATIVE
MAKE OR BREAK DIMENSION: PROJECT TRACKER & PERFORMANCE DASHBOARD
Priority Test DescriptionRequestors/Key
Stakeholders
Type of
ChangeHypotheses
How did we
arrive at this
hypotheses
Where will
the Test
happen?
Target
Audience
1
Remove Ad
banner on Yahoo
home page
User Experience Prominence
Removing Ad
banners would
reduce
distraction and
focus users to
CTA
Product/Design
JudgementHome Page All Consumers
Standard Test
Plan Document
Ready
#Test Cells
#Days needed for the
Test to run tor
statistical significant
sample
Design
Ready?
Specific
Technical
Requirements?
Estimated Tech
Effort/Cost
(USD)
Overall Test Cost
(USD)
Yes 2 40 Yes
Test Details
Other details from the Test
ILLUSTRATIVE
MAKE OR BREAK DIMENSION: PROJECT TRACKER & PERFORMANCE DASHBOARD CONTD
Intended for Knowledge Sharing only
ILLUSTRATIVE
Primary Metrics Secondary MetricsEstimated Benefit
(USD)Click Through Rate Net Promoter Score Repeat VisitsCustomer Lifetime
Value
x% y% z% a%
Expected Impact from the Test
Primary Metrics Secondary MetricsEstimated Benefit
(USD)Click Through Rate Net Promoter Score Repeat VisitsCustomer Lifetime
Value
x% y% z% a%
Actual Impact from the Test
& COMMUNICATION READOUT AT REGULAR CADENCE!
Objective
Understand if removing Ad banner on home page improves click through rate on articles and increases consumer
satisfaction
0%
20%
40%
60%
80%
100%
120%
0%
2%
4%
6%
8%
10%
12%
14%
16%
18%
Delt
a b
etw
ee
n T
es
t &
Co
ntr
ol
Te
st/
Co
ntr
ol V
alu
es
Test metrics - Click through Rate
Delta Test Control
Key Findings
1. Removing the banner increased CTR by '100%' and NPS by 20 points '. It translates to $40 M in Lifetime Value impact.
2. All the above lifts are statistically significant at 90% confidence level. These lifts were also consistent over two weeks
time window.
Sl.No.
1
2
3
5
Performance data Time window: Apr 1, 1980 to Apr 14, 1980
ILLUSTRATIVE
THINGS WE WATCH OUT FOR
• Engineering overheads – every time a new flow needs to be introduced or any major addition to the
experience, new development is required. It has to go through Standard engineering prioritization route unless a
SWAT team is dedicated to it.
• Tricky QA situations – QA team should be trained to handle A/B Testing scenarios and use cases; Integration
with automated QA tools. Security and FE load failure considerations apart from standard checks.
• Operational excellence requirements – Testing of the Tests in Sandbox, Staging and Live Site Testing areas.
End to End Dry runs mandatory being launching the tests.
• Analytical nuances – Experiment Design supreme need! External factors can easily invalidate A/B Testing.
Sample fragmentation with increasing #tests and complexity; Need for Universal Control; Impact should be
checked for significance over time.
• Data needs – Reliable instrumentation, Testing Tool JavaScript put in right place, with minimal overhead
performance impact, integration with Web Analytics tool, Data feed with ability to tie with other data sources
(for deep dives).
• Branding Guidelines – Don’t overwhelm and confuse users in quest for multiple and complex tests; Standardize
but customize experience across various channels and platforms; Soft launches should be as much avoided as
possible.
• Proactive internal communication, specifically to client facing teams.
• Strategic Decisions – Some changes have to go in irrespective of A/B Testing findings, the question would be
how to make it happen right? This is gradual ramp, progressive learning and iterative improvements – a collection
of A/B Tests and not one off big one.
…A/B Testing can never be a failure, by definition it is a learning on whether the change was well
received by the user or not that informs the next steps
Intended for Knowledge Sharing only
Quick recap of what it is
Intended for Knowledge Sharing only
Discussion items
27
QUESTIONS FOR THE AUDIENCE
Where are you in the Testing Maturity Curve?
What were your biggest bottlenecks and how did you solve them?
Were you successful in up-leveling the conversation in your organization?
How did you crack the Resourcing & Funding problem?
What are the things that worked best for you in your journey?
How did you protect Testing resources from being used up for CMS or Bug Fixes?
How did you manage the nuance between Learning and Business Objectives?
How did you convince the organization to use Testing as driver of accountability
but also not get dragged into for political issues?
Intended for Knowledge Sharing only
Intended for Knowledge Sharing only
Quick recap of what it is
Intended for Knowledge Sharing only
The parting words…
29
KEY TAKEAWAYS
Intended for Knowledge Sharing only
An advanced Experimentation program is hallmark of a “Data Driven
Decision Making Culture” of accountability & transparency
Benefit from Experimentation is best realized when it’s anchored to
Strategic goals and is driven with insights from Analytics and Research
Mature organization leverage Algorithmic Test Management framework to
achieve scalability and efficiency at Optimal Program RoI levels
Organizations with a disciplined Experimentation culture within the DNA
are poised to reap benefits of higher accountability, focus on business
performance and optimized Customer Experience Management
Testing Program is a high reward but high investment-high political risk
function and an executive leadership & support are imperative
Intended for Knowledge Sharing only
Quick recap of what it is
Intended for Knowledge Sharing only
Appendix
31
THANK YOU!
Would love to hear from you on any of the following forums…
https://twitter.com/decisions_2_0
http://www.slideshare.net/RamkumarRavichandran
https://www.youtube.com/channel/UCODSVC0WQws607clv0k8mQA/videos
http://www.odbms.org/2015/01/ramkumar-ravichandran-visa/
https://www.linkedin.com/pub/ramkumar-ravichandran/10/545/67a
RAMKUMAR RAVICHANDRAN
ROGER CHANG
https://www.linkedin.com/in/rogervchang/