BETFAIR'S SITE REBUILD: FAST - WE PROMISE Martin Anderson LJC Open Conference 2011

2

THE EXCHANGE

3

OUR PRODUCTS

4

3.7M+ registered customers

140 locations

17 languages

£300M funds on deposit

30,000 bets placed one minute

88,000 http requests per second

INTERESTING STATS

5

WHAT’S THE PROBLEM?

“during the world cup the site has run like treacle”

“the site is frequently slow and freezes too often”

“very slow and glitchy”

“I find Betfair slow”

We lost focus on our website

6

http://bit.ly/bf-cc OUR CUSTOMER COMMITMENT

7

PERFORMANCE – 95 PECENTILE

8

OPTIONS?

•  Fix •  Rebuild

•  We chose to rebuild with a new architecture

•  Started earlier this year

9

It Works

It’s Fast

It’s Useful

It’s Cool

HIERARCHY OF WEBSITE NEEDS

10

FIVE FUNDAMENTALS OF A WORLD CLASS WEB SITE

•  Operational Monitoring •  Performance •  Customer Analytics •  SEO •  Experimentation Framework

11

DOT COM ERA ARCHITECTURE

Single component responsible for data

access, business logic and presentation

12

CURRENT ARCHITECTURE

Data Services

Lightweight web servers for cached

HTML and JS

Heavy use of client-side rendering with

JavaScript

13

NEXT GENERATION ARCHITECTURE

Data Services

Server-side rendering using Java

No client-side rendering only updates

via JavaScript

14

TECHNOLOGY STACK

•  Java, JavaScript (YUI), FreeMarker, Spring 3.0, Tomcat 7

•  TestNG, Mockito, Groovy, Selenium2, Jasmine, PhantomJS

15

WHY CHOOSE JAVA?

•  Good knowledge already at Betfair •  Real concurrency – great for heavy server

workload •  Large Community •  Great Toolset •  Operations Teams understand Java – stats,

GC logs, deployment process

16

JAVA IN THE WEB TIER

•  Java is obviously great at server tasks but does it work well in the web tier?

•  Can it be as flexible as Ruby or Python?

17

MODULE CONFIGURATION FRAMEWORK

•  Deals with the impendence mismatch between the web and statically typed languages

•  Hugely flexible •  Easy to work with heterogeneous skillsets

18

MODULE CONFIGURATION FRAMEWORK

•  All pages are an aggregation of modules in a layout

•  Modules are the smallest measure of functionality on a page

•  Modules can live within other modules •  Configuration controls the service calls and

the rendered modules

19

MODULE CONFIGURATION FRAMEWORK

20

CONCURRENCY MODEL

dispatch service calls

controller’s job done

rendering starts early

21

PERFORMANCE OPTIMIZATIONS

•  Reduce HTTP requests • Minify and compress assets (WRO4J) •  Split requests across domains • Use cookie-free CDN hostnames •  Flush the buffer early • Defer loading of content •  Etc…

22

CUSTOMER ANALYTICS

• Omniture Tagging •  Server side clickstream

•  Able to generate heatmaps of site usage

23

SEARCH ENGINE OPTIMISATION

•  Single HTML response •  “Pretty” URLs •  Tasty bot fodder • No redirects • No duplications •  Crafted content

24

EXPERIMENTATION FRAMEWORK

•  An extension of the MCF •  Allows A/B or Multivariate testing •  Allows business to make decisions based on empirical data

• Winning experiments can be rolled out without a redeployment

25

CONTINUOUS DELIVERY

•  Its not about changing the product daily •  Its about confidence in the quality of your

product •  Its about eliminating waste from your

development cycle

26

CHALLENGES

•  Automation is key •  Branch in code •  Feature Toggles •  Technology only takes you so far; it’s a

cultural shift •  “Developers aren’t done until the tests

pass”

27

28

OPERATIONAL MONITORING

•  Analytics •  Trending •  Fault-detection / alerting •  Capacity planning

29

PERFORMANCE MEASUREMENT

30

LOG MINING

Access logs: User agents, response times, payload sizes Performance logs: Timings of code paths and dependent service calls Correlation IDs to trace requests across tiers

31

GRAPHS

32

SERVER-SIDE WATERFALL GRAPH

33

FLUSHING THE BUFFER

Networking device thwarting flush() code Magic setting: httpcompressonpush=1

34

RESULT

35

SO WHAT CHANGED?

36

FEEDBACK

Customer: “Faster”

Customer: “Seems faster and more user friendly”

Customer: “page loaded quicker”

Ops Guy: “Our current down time is zero!”

37

RESULTS

•  Sessions that bet: up 2 percentage points

•  Bounce rate: improved 40% •  Page views: 2x •  1% opt-out rate

38

WHAT HAVE WE LEARNED?

•  Performance improvements are tangible • Measurably improves the bottom line •  Operational Monitoring and Customer

Analytics provides essential visibility •  There’s lots more to do but we now have the

platform to do this

39

THANK YOU

Email: martin.anderson@betfair.com Twitter: @mdjanderson Blog: http://bloodredsun.com/ Betfair Blog: http://views.betfair.com/

40