a/b testing at the edge
TRANSCRIPT
A/B testing at the edgeChris Jackel | Systems Engineer, Fastly
Complexity
TooBasic
Complexity
TooBasic
TooCode-y
Complexity
TooBasic
Complexity
Sweet spot for maximal dissatisfaction
TooCode-y
Why, How, and More How
• Why do A/B testing?• How to implement with Fastly• Non-traditional uses
Why
• Science!
https://blog.optimizely.com/2010/11/29/
What We Need
• ‘A’ and a ‘B’
What We Need
• ‘A’ and a ‘B’
• Method of (persistent) segmentation
What We Need
• ‘A’ and a ‘B’
• Method of (persistent) segmentation
• Keeping things separate (vary)
What We Need
• ‘A’ and a ‘B’
• Method of (persistent) segmentation
• Keeping things separate (vary)
Web ServerClients
GET /
Fastly
Segmentation
sub vcl_recv {
if (randombool(5,100)) { set req.http.X-FastAB = "B"; } else { set req.http.X-FastAB = "A"; }
}
Segmentation
sub vcl_recv {
if (randombool(5,100)) { set req.http.X-FastAB = "B"; } else { set req.http.X-FastAB = "A"; }
}
Web ServerClients Fastly
GET / X-FastAB: A (95%/5%)
Web ServerClients Fastly
200 OK (version ‘A’)
Web ServerClients Fastly
200 OK (version ‘A’)
Segmentation
sub vcl_deliver {
add resp.http.Set-Cookie = "FastAB=" req.http.X-FastAB ”; expires=Thu, 20-Jul-17 19:47:08;”;
return(deliver); }
Segmentation
sub vcl_deliver {
if (!req.http.Cookie:FastAB){
add resp.http.Set-Cookie = "FastAB=" req.http.X-FastAB ”; expires=Thu, 20-Jul-17 19:47:08;”;
}
return(deliver); }
Cookie?No Yes
Segment A/B
Set Header to Server
Set Cookie on Response
Set Header to Server = Cookie
Segmentation
sub vcl_recv {
if (randombool(5,100)) { set req.http.X-FastAB = "B"; } else { set req.http.X-FastAB = "A"; }
}
Segmentation
sub vcl_recv { if (!req.http.Cookie:FastAB) {
if (randombool(5,100)) { set req.http.X-FastAB = "B"; } else { set req.http.X-FastAB = "A"; } }
}
Segmentation
sub vcl_recv { if (!req.http.Cookie:FastAB) {
if (randombool(5,100)) { set req.http.X-FastAB = "B"; } else {
set req.http.X-FastAB = "A"; } } else {
set req.http.X-FastAB = req.http.Cookie:FastAB;
} }
What We Need
• ‘A’ and a ‘B’
• Method of (persistent) segmentation
• Keeping things separate (vary)
HTTP Vary
HTTP Vary
- The chef (or server) tells us what is important information (e.g. flavor)
- The customer (or browser) tells us the flavor
- Max of 200 variations!
- Otherwise change the hash
- Don’t vary on too big a set (e.g. user-agent)
200 OK
200 OK (mild)
WTF?
200 OK Vary: X-Flavor
HTTP Vary
if (beresp.http.Vary) { set beresp.http.Vary = beresp.http.Vary ", X-FastAB"; } else { set beresp.http.Vary = "X-FastAB"; }
HTTP Vary
if (beresp.http.Vary) { set beresp.http.Vary = beresp.http.Vary ", X-FastAB"; } else { set beresp.http.Vary = "X-FastAB"; }
# e.g. Vary: Accept-Encoding, X-FastAB
HTTP Vary
• What is a hash?
HTTP Vary
• What is a hash?
{ set req.hash += req.url; set req.hash += req.http.host; set req.hash += "#####GENERATION#####"; return (hash); }
Other Topics
• Backend uses
• Updating weights with Edge Dictionaries
• Cookie-less options
Backend AB
- DR Testing
Backend AB
if (randombool(5,100)) { set req.backend = backend_b; } else { set req.backend = backend_a; }
Backend AB
- DR Testing
- CMS Migration
Backend AB
sub vcl_recv {
if (randombool(5,100)) { set req.url = “/v2“ req.url; }
}
Edge Dictionary
Change weights via API
Edge Dictionary
Change weights via API
table page_segments {
"/" : “5”, # percentage ‘b’ ”/page1" : "30", "/page2" : “40",
}
Edge Dictionary
Change weights via API
if (randombool(5,100)) { set req.http.X-FastAB = "B"; } else { set req.http.X-FastAB = "A"; }
Edge Dictionary
set req.http.X-Weight = table.lookup(page_segments, req.url.path);
if (randombool(5,100)) { set req.http.X-FastAB = "B"; } else { set req.http.X-FastAB = "A"; }
Edge Dictionaryset req.http.X-Weight = table.lookup(page_segments, req.url.path);
if (randombool(req.http.X-Weight,100)) { set req.http.X-FastAB = "B"; } else { set req.http.X-FastAB = "A"; }
Edge Dictionaryset req.http.X-Weight = table.lookup(page_segments, req.url.path);
if (randombool(std.atoi(req.http.X- Weight),100)) { set req.http.X-FastAB = "B"; } else { set req.http.X-FastAB = "A"; }
Cookie-less
if (randombool(5,100)) { set req.http.X-FastAB = "B"; } else { set req.http.X-FastAB = “A"; }
Cookie-less
if (randombool_seeded(5,100,seed))) { set req.http.X-FastAB = "B"; } else { set req.http.X-FastAB = “A"; }
Cookie-less
set req.http.X-ClientIDHash = digest.hash_md5(client.ip req.http.User-Agent)
# e.g. 3aaf91f17e7c63b07f8490ab242f9335
if (randombool_seeded(5,100,seed))) { set req.http.X-FastAB = "B"; } else { set req.http.X-FastAB = “A"; }
Cookie-less
set req.http.X-ClientIDHash = digest.hash_md5(client.ip req.http.User-Agent)
set req.http.X-ClientID = std.strtol(req.http.X-ClientIDHash,16)
# e.g. 9223372036854775807
if (randombool_seeded(5,100,seed))) { set req.http.X-FastAB = "B"; } else { set req.http.X-FastAB = “A"; }
Cookie-less
set req.http.X-ClientIDHash = digest.hash_md5(client.ip req.http.User-Agent)
set req.http.X-ClientID = std.strtol(req.http.X-ClientIDHash,16)
# e.g. 9223372036854775807
if (randombool_seeded(5,100,std.atoi(req.http.X-ClientID))) { set req.http.X-FastAB = "B"; } else { set req.http.X-FastAB = “A"; }
Cookie-less
if (randombool_seeded(5,100,std.atoi(req.http.X-ClientID))) { set req.http.X-FastAB = "B"; } else { set req.http.X-FastAB = “A"; }
Still the seed
Cookie-less
if (randombool_seeded(5,100,std.atoi(req.http.X-ClientID))) { set req.http.X-FastAB = "B"; } else { set req.http.X-FastAB = “A"; }
Still the seed
The data must flow
Thanks!