mvt across multiple brands: one approach to successful testing, webtrends
DESCRIPTION
TRANSCRIPT
![Page 1: MVT Across Multiple Brands: One approach to successful testing, Webtrends](https://reader038.vdocuments.site/reader038/viewer/2022102814/54c0fa214a7959a8138b45ac/html5/thumbnails/1.jpg)
© 2011 Webtrends, All Rights Reserved.
MVT Across Multiple Brands: One approach to successful testing
Martin SheerinHead of Online Product Development, Phones4UMichael Schirrmacher UK Sales Director, Webtrends
Webtrends |
![Page 2: MVT Across Multiple Brands: One approach to successful testing, Webtrends](https://reader038.vdocuments.site/reader038/viewer/2022102814/54c0fa214a7959a8138b45ac/html5/thumbnails/2.jpg)
© 2011 Webtrends, All Rights Reserved. |2
Why should we Optimize?
#1. Improve Customer Experience
#2. Improve Conversion & Revenue
#3. Deepen Visitor Engagement
![Page 3: MVT Across Multiple Brands: One approach to successful testing, Webtrends](https://reader038.vdocuments.site/reader038/viewer/2022102814/54c0fa214a7959a8138b45ac/html5/thumbnails/3.jpg)
© 2011 Webtrends, All Rights Reserved. |3
![Page 4: MVT Across Multiple Brands: One approach to successful testing, Webtrends](https://reader038.vdocuments.site/reader038/viewer/2022102814/54c0fa214a7959a8138b45ac/html5/thumbnails/4.jpg)
© 2011 Webtrends, All Rights Reserved. |4
Where do I start?
Analytics
Identify and quantify opportunities through
insight
Optimization
Test, segment and target to improve performance
![Page 5: MVT Across Multiple Brands: One approach to successful testing, Webtrends](https://reader038.vdocuments.site/reader038/viewer/2022102814/54c0fa214a7959a8138b45ac/html5/thumbnails/5.jpg)
© 2011 Webtrends, All Rights Reserved. |5
What Optimisation Involves?
![Page 6: MVT Across Multiple Brands: One approach to successful testing, Webtrends](https://reader038.vdocuments.site/reader038/viewer/2022102814/54c0fa214a7959a8138b45ac/html5/thumbnails/6.jpg)
© 2011 Webtrends, All Rights Reserved. |6
Full Factorial….Fractional factorial….?
“ok this looks complex”
![Page 7: MVT Across Multiple Brands: One approach to successful testing, Webtrends](https://reader038.vdocuments.site/reader038/viewer/2022102814/54c0fa214a7959a8138b45ac/html5/thumbnails/7.jpg)
© 2011 Webtrends, All Rights Reserved. |7
So, for now…a managed service is still more popular
![Page 8: MVT Across Multiple Brands: One approach to successful testing, Webtrends](https://reader038.vdocuments.site/reader038/viewer/2022102814/54c0fa214a7959a8138b45ac/html5/thumbnails/8.jpg)
© 2011 Webtrends, All Rights Reserved. |8
Optimization Maturity Model
Conversions
• Web Analytics• Measurement • No Testing• Ad Network
Targeting
• A/B/n Testing• Basic MVT -1-3
areas• 1 Element Targeting• 1-Project at a Time
• More Advanced MVT• In-session
Segmentation• Multi Element
Targeting • Frequent
• Personalization• Integrated
Optimization• External Segment
Use• Social & Multi-
Channel• On-going
Optimization Maturity Phases
“Crawl ” Phase “Walk” Phase “Run” PhaseOperational Reporting
![Page 9: MVT Across Multiple Brands: One approach to successful testing, Webtrends](https://reader038.vdocuments.site/reader038/viewer/2022102814/54c0fa214a7959a8138b45ac/html5/thumbnails/9.jpg)
© 2011 Webtrends, All Rights Reserved. |9
Multivariate Test
£
Control
![Page 10: MVT Across Multiple Brands: One approach to successful testing, Webtrends](https://reader038.vdocuments.site/reader038/viewer/2022102814/54c0fa214a7959a8138b45ac/html5/thumbnails/10.jpg)
© 2011 Webtrends, All Rights Reserved. |10
Multivariate Test
Challenger
£
Control
![Page 11: MVT Across Multiple Brands: One approach to successful testing, Webtrends](https://reader038.vdocuments.site/reader038/viewer/2022102814/54c0fa214a7959a8138b45ac/html5/thumbnails/11.jpg)
© 2011 Webtrends, All Rights Reserved. |11
Multivariate Test
+£3.2m Lift in special offers sales in UK
![Page 12: MVT Across Multiple Brands: One approach to successful testing, Webtrends](https://reader038.vdocuments.site/reader038/viewer/2022102814/54c0fa214a7959a8138b45ac/html5/thumbnails/12.jpg)
Martin SheerinHead of Online Product Development Phones 4U
@martinsheerin1
linkedin.com/in/martinsheerin1
![Page 13: MVT Across Multiple Brands: One approach to successful testing, Webtrends](https://reader038.vdocuments.site/reader038/viewer/2022102814/54c0fa214a7959a8138b45ac/html5/thumbnails/13.jpg)
Phones 4u
• Multi-channel retailer• Over 600 stores• Contact centre for sales and service• Website: www.phones4u.co.uk• Mobile site: m.phones4u.co.uk
• Online only brand:– www.dialaphone.co.uk– m.dialaphone.co.uk
![Page 14: MVT Across Multiple Brands: One approach to successful testing, Webtrends](https://reader038.vdocuments.site/reader038/viewer/2022102814/54c0fa214a7959a8138b45ac/html5/thumbnails/14.jpg)
Why should you MVT?
Increase Conversion
![Page 15: MVT Across Multiple Brands: One approach to successful testing, Webtrends](https://reader038.vdocuments.site/reader038/viewer/2022102814/54c0fa214a7959a8138b45ac/html5/thumbnails/15.jpg)
Why not MVT yourself instead of using a partner?• Time• Skills• Ensuring that a test is statistically significant• Ensures the result is true and not biased to an internal individuals
viewpoint
• You can do this yourself:
![Page 16: MVT Across Multiple Brands: One approach to successful testing, Webtrends](https://reader038.vdocuments.site/reader038/viewer/2022102814/54c0fa214a7959a8138b45ac/html5/thumbnails/16.jpg)
What to look for in an MVT provider?
Business understandi
ng
eCommerce insight
Business alignment
Ease of reporting
![Page 17: MVT Across Multiple Brands: One approach to successful testing, Webtrends](https://reader038.vdocuments.site/reader038/viewer/2022102814/54c0fa214a7959a8138b45ac/html5/thumbnails/17.jpg)
Our Approach
• Test simultaneously on both sites
• Use results of test on one site and apply to the other (if its possible)
• Be happy to repeat tests annually (or more regularly if you deem necessary)
![Page 18: MVT Across Multiple Brands: One approach to successful testing, Webtrends](https://reader038.vdocuments.site/reader038/viewer/2022102814/54c0fa214a7959a8138b45ac/html5/thumbnails/18.jpg)
Our Test Successes
![Page 19: MVT Across Multiple Brands: One approach to successful testing, Webtrends](https://reader038.vdocuments.site/reader038/viewer/2022102814/54c0fa214a7959a8138b45ac/html5/thumbnails/19.jpg)
Free Gifts
Op
tim
al
Co
ntr
ol P
ag
e
6.27%lift
![Page 20: MVT Across Multiple Brands: One approach to successful testing, Webtrends](https://reader038.vdocuments.site/reader038/viewer/2022102814/54c0fa214a7959a8138b45ac/html5/thumbnails/20.jpg)
Customer Behaviour
Research
Choose Handset
Allowance (Lifestyle)
Select Package
Purchase
Phones 4u
Dialaphone
![Page 21: MVT Across Multiple Brands: One approach to successful testing, Webtrends](https://reader038.vdocuments.site/reader038/viewer/2022102814/54c0fa214a7959a8138b45ac/html5/thumbnails/21.jpg)
Phones 4u Review Tab
Control
Winner
+2.99%
![Page 22: MVT Across Multiple Brands: One approach to successful testing, Webtrends](https://reader038.vdocuments.site/reader038/viewer/2022102814/54c0fa214a7959a8138b45ac/html5/thumbnails/22.jpg)
Search Results
+16.56%
![Page 23: MVT Across Multiple Brands: One approach to successful testing, Webtrends](https://reader038.vdocuments.site/reader038/viewer/2022102814/54c0fa214a7959a8138b45ac/html5/thumbnails/23.jpg)
Simple and Usable
Source: Simple and Usable Web, Mobile, and Interaction Design (Voices That Matter) - Giles Colborne (CX Partners) (2010)
![Page 24: MVT Across Multiple Brands: One approach to successful testing, Webtrends](https://reader038.vdocuments.site/reader038/viewer/2022102814/54c0fa214a7959a8138b45ac/html5/thumbnails/24.jpg)
Search Results
Op
tim
al
Co
ntr
ol P
ag
e
Experiment Page Lift
Control ---
Optimal +16.56%
+16.56%
![Page 25: MVT Across Multiple Brands: One approach to successful testing, Webtrends](https://reader038.vdocuments.site/reader038/viewer/2022102814/54c0fa214a7959a8138b45ac/html5/thumbnails/25.jpg)
Upgrades - Variants
![Page 26: MVT Across Multiple Brands: One approach to successful testing, Webtrends](https://reader038.vdocuments.site/reader038/viewer/2022102814/54c0fa214a7959a8138b45ac/html5/thumbnails/26.jpg)
Op
tim
al
Co
ntr
ol P
ag
e
Experiment Page Lift
Control ---
Optimal +81.64%
+81.64%Lift at next step
![Page 27: MVT Across Multiple Brands: One approach to successful testing, Webtrends](https://reader038.vdocuments.site/reader038/viewer/2022102814/54c0fa214a7959a8138b45ac/html5/thumbnails/27.jpg)
Taking this finding we applied it to Phones 4u
Optimal
Lev
el 2
- g
rey
bac
kgro
un
d
and
up
gra
de
pro
mis
e
Lev
el 3
–
wh
ite
bac
kgro
un
d
and
up
gra
de
pro
mis
e
Lev
el 4
–
gre
y b
ackg
rou
nd
Lev
el 5
–
wh
ite
bac
kgro
un
d
con
tro
l
![Page 28: MVT Across Multiple Brands: One approach to successful testing, Webtrends](https://reader038.vdocuments.site/reader038/viewer/2022102814/54c0fa214a7959a8138b45ac/html5/thumbnails/28.jpg)
Phones 4u Winning Variant
Op
tim
al
Co
ntr
ol P
ag
e
Experiment Page Lift
Control ---
Optimal +70.64%
+70.64 %Lift at next step
![Page 29: MVT Across Multiple Brands: One approach to successful testing, Webtrends](https://reader038.vdocuments.site/reader038/viewer/2022102814/54c0fa214a7959a8138b45ac/html5/thumbnails/29.jpg)
Not satisfied we went with a hunch…
Op
tim
al
Co
ntr
ol P
ag
e
Experiment Page Lift
Control ---
Optimal +0.89%
+0.89%
![Page 30: MVT Across Multiple Brands: One approach to successful testing, Webtrends](https://reader038.vdocuments.site/reader038/viewer/2022102814/54c0fa214a7959a8138b45ac/html5/thumbnails/30.jpg)
Moving on with the upgrades test
![Page 31: MVT Across Multiple Brands: One approach to successful testing, Webtrends](https://reader038.vdocuments.site/reader038/viewer/2022102814/54c0fa214a7959a8138b45ac/html5/thumbnails/31.jpg)
Multi-device testingC
on
tro
l Pa
ge
![Page 32: MVT Across Multiple Brands: One approach to successful testing, Webtrends](https://reader038.vdocuments.site/reader038/viewer/2022102814/54c0fa214a7959a8138b45ac/html5/thumbnails/32.jpg)
Applying this idea to Phones 4u
![Page 33: MVT Across Multiple Brands: One approach to successful testing, Webtrends](https://reader038.vdocuments.site/reader038/viewer/2022102814/54c0fa214a7959a8138b45ac/html5/thumbnails/33.jpg)
All Traffic (non-iPad)C
on
tro
l Pa
ge
Op
tim
al
Experiment Page Lift
Control ---
Optimal +4.11%
Optimal
![Page 34: MVT Across Multiple Brands: One approach to successful testing, Webtrends](https://reader038.vdocuments.site/reader038/viewer/2022102814/54c0fa214a7959a8138b45ac/html5/thumbnails/34.jpg)
iPad VersionC
on
tro
l Pa
ge
Optimal
Experiment Page Lift
Control ---
Optimal +4.27%
Op
tim
al
![Page 35: MVT Across Multiple Brands: One approach to successful testing, Webtrends](https://reader038.vdocuments.site/reader038/viewer/2022102814/54c0fa214a7959a8138b45ac/html5/thumbnails/35.jpg)
Summary
Define what success
looks like
Apply common
sense
Only test what you are willing
to put live
Don’t be satisfied that the result of one test cannot
be improved upon
Re-test if you think its
necessary
Sometimes the simplest things give the biggest
results
Geo targeting
Persona driven
segmentation
Touch commerce
(Multi Device)
![Page 36: MVT Across Multiple Brands: One approach to successful testing, Webtrends](https://reader038.vdocuments.site/reader038/viewer/2022102814/54c0fa214a7959a8138b45ac/html5/thumbnails/36.jpg)
Martin SheerinHead of Online Product Development Phones 4U
@martinsheerin1
linkedin.com/in/martinsheerin1