somesql at skyscanner - scaling in a changing world of databases and hardware
DESCRIPTION
Slides from Alistair Hann's presentation at 2014 All Your Base Conference in Oxford, UK Synopsis: Skyscanner performs 200 million searches every month, and generated $7bn of downstream revenue last year. Scaling from zero brought many challenges, in the context of a continually growing variety of databases and hardware. Alistair will talk about how that rapid change has shaped Skyscanner’s data architecture and how they moved from just using SQL Server to a range of hardware and data technologies (Postgres, Couchbase, ElasticSearch, Hadoop). (Unfortunately the animations were destroyed by the upload)TRANSCRIPT
![Page 1: SomeSQL at Skyscanner - Scaling in a changing world of databases and hardware](https://reader033.vdocuments.site/reader033/viewer/2022061206/5476d025b4af9f2f628b45cd/html5/thumbnails/1.jpg)
SomeSQLScaling in a changing world of databases and hardware
Alistair HannCTO, Skyscanner
![Page 2: SomeSQL at Skyscanner - Scaling in a changing world of databases and hardware](https://reader033.vdocuments.site/reader033/viewer/2022061206/5476d025b4af9f2f628b45cd/html5/thumbnails/2.jpg)
Buzzwords
![Page 3: SomeSQL at Skyscanner - Scaling in a changing world of databases and hardware](https://reader033.vdocuments.site/reader033/viewer/2022061206/5476d025b4af9f2f628b45cd/html5/thumbnails/3.jpg)
Web 2.0
![Page 4: SomeSQL at Skyscanner - Scaling in a changing world of databases and hardware](https://reader033.vdocuments.site/reader033/viewer/2022061206/5476d025b4af9f2f628b45cd/html5/thumbnails/4.jpg)
Year of Mobile
![Page 5: SomeSQL at Skyscanner - Scaling in a changing world of databases and hardware](https://reader033.vdocuments.site/reader033/viewer/2022061206/5476d025b4af9f2f628b45cd/html5/thumbnails/5.jpg)
Big Data
![Page 6: SomeSQL at Skyscanner - Scaling in a changing world of databases and hardware](https://reader033.vdocuments.site/reader033/viewer/2022061206/5476d025b4af9f2f628b45cd/html5/thumbnails/6.jpg)
NoSQL
![Page 7: SomeSQL at Skyscanner - Scaling in a changing world of databases and hardware](https://reader033.vdocuments.site/reader033/viewer/2022061206/5476d025b4af9f2f628b45cd/html5/thumbnails/7.jpg)
Scaling the live pricing cache
![Page 8: SomeSQL at Skyscanner - Scaling in a changing world of databases and hardware](https://reader033.vdocuments.site/reader033/viewer/2022061206/5476d025b4af9f2f628b45cd/html5/thumbnails/8.jpg)
![Page 9: SomeSQL at Skyscanner - Scaling in a changing world of databases and hardware](https://reader033.vdocuments.site/reader033/viewer/2022061206/5476d025b4af9f2f628b45cd/html5/thumbnails/9.jpg)
Website Native Apps APIs and White Labels
Traditional Airlines Budget Airlines Online Travel Agencies
![Page 10: SomeSQL at Skyscanner - Scaling in a changing world of databases and hardware](https://reader033.vdocuments.site/reader033/viewer/2022061206/5476d025b4af9f2f628b45cd/html5/thumbnails/10.jpg)
![Page 11: SomeSQL at Skyscanner - Scaling in a changing world of databases and hardware](https://reader033.vdocuments.site/reader033/viewer/2022061206/5476d025b4af9f2f628b45cd/html5/thumbnails/11.jpg)
![Page 12: SomeSQL at Skyscanner - Scaling in a changing world of databases and hardware](https://reader033.vdocuments.site/reader033/viewer/2022061206/5476d025b4af9f2f628b45cd/html5/thumbnails/12.jpg)
Prices +Timetables
Data Collection Services
1) Which websites should we show?
2) What prices do we already have
cached?
3) Live update what we still
need.
4) Clean up and save the new data
5) Return the prices to the user.
Live Pricing Service
![Page 13: SomeSQL at Skyscanner - Scaling in a changing world of databases and hardware](https://reader033.vdocuments.site/reader033/viewer/2022061206/5476d025b4af9f2f628b45cd/html5/thumbnails/13.jpg)
Live Pricing Service
![Page 14: SomeSQL at Skyscanner - Scaling in a changing world of databases and hardware](https://reader033.vdocuments.site/reader033/viewer/2022061206/5476d025b4af9f2f628b45cd/html5/thumbnails/14.jpg)
Live Pricing Service
Cached Prices (key/value)
![Page 15: SomeSQL at Skyscanner - Scaling in a changing world of databases and hardware](https://reader033.vdocuments.site/reader033/viewer/2022061206/5476d025b4af9f2f628b45cd/html5/thumbnails/15.jpg)
2 bn itineraries and quotes
270 GB
table
250 GB
indices
2000 quotes per second
![Page 16: SomeSQL at Skyscanner - Scaling in a changing world of databases and hardware](https://reader033.vdocuments.site/reader033/viewer/2022061206/5476d025b4af9f2f628b45cd/html5/thumbnails/16.jpg)
What we really needed
Consistency
Horizontal Scaling
Elasticity
Persistence
Speed
Resilience
Simplicity
![Page 17: SomeSQL at Skyscanner - Scaling in a changing world of databases and hardware](https://reader033.vdocuments.site/reader033/viewer/2022061206/5476d025b4af9f2f628b45cd/html5/thumbnails/17.jpg)
Live Pricing Service
Cached Prices (key/value)
![Page 18: SomeSQL at Skyscanner - Scaling in a changing world of databases and hardware](https://reader033.vdocuments.site/reader033/viewer/2022061206/5476d025b4af9f2f628b45cd/html5/thumbnails/18.jpg)
![Page 19: SomeSQL at Skyscanner - Scaling in a changing world of databases and hardware](https://reader033.vdocuments.site/reader033/viewer/2022061206/5476d025b4af9f2f628b45cd/html5/thumbnails/19.jpg)
![Page 20: SomeSQL at Skyscanner - Scaling in a changing world of databases and hardware](https://reader033.vdocuments.site/reader033/viewer/2022061206/5476d025b4af9f2f628b45cd/html5/thumbnails/20.jpg)
![Page 21: SomeSQL at Skyscanner - Scaling in a changing world of databases and hardware](https://reader033.vdocuments.site/reader033/viewer/2022061206/5476d025b4af9f2f628b45cd/html5/thumbnails/21.jpg)
Beyond key value
![Page 22: SomeSQL at Skyscanner - Scaling in a changing world of databases and hardware](https://reader033.vdocuments.site/reader033/viewer/2022061206/5476d025b4af9f2f628b45cd/html5/thumbnails/22.jpg)
![Page 23: SomeSQL at Skyscanner - Scaling in a changing world of databases and hardware](https://reader033.vdocuments.site/reader033/viewer/2022061206/5476d025b4af9f2f628b45cd/html5/thumbnails/23.jpg)
Couchbase – Map Reduce Views
{ "website": { "published": true, "id": "affd", ... }, "office_id": "1", "city_id": "AUHA", "raw_data": [...] "address": "closing_time": "00:00", "routenodeid": "9618", "type": "office“}
![Page 24: SomeSQL at Skyscanner - Scaling in a changing world of databases and hardware](https://reader033.vdocuments.site/reader033/viewer/2022061206/5476d025b4af9f2f628b45cd/html5/thumbnails/24.jpg)
What about the hardware?
![Page 25: SomeSQL at Skyscanner - Scaling in a changing world of databases and hardware](https://reader033.vdocuments.site/reader033/viewer/2022061206/5476d025b4af9f2f628b45cd/html5/thumbnails/25.jpg)
Disk for VMs
c.f. 250,000 iops Fusion I/O
Standard $0.03 / GBGlacier $0.01 / GB
![Page 26: SomeSQL at Skyscanner - Scaling in a changing world of databases and hardware](https://reader033.vdocuments.site/reader033/viewer/2022061206/5476d025b4af9f2f628b45cd/html5/thumbnails/26.jpg)
Quote Bus staging
UK1
Thrift
long-term archive
GZIP
queryablehierarchical
LZO
queryableflat
filter
Loader
GZIP
Quote Bus
UK2
Thrift
Loader
![Page 27: SomeSQL at Skyscanner - Scaling in a changing world of databases and hardware](https://reader033.vdocuments.site/reader033/viewer/2022061206/5476d025b4af9f2f628b45cd/html5/thumbnails/27.jpg)
hierarchical
flat
Hadoop clusteror
Elastic MapReduce
analystsquery
load
analytical tools
feed
export
![Page 28: SomeSQL at Skyscanner - Scaling in a changing world of databases and hardware](https://reader033.vdocuments.site/reader033/viewer/2022061206/5476d025b4af9f2f628b45cd/html5/thumbnails/28.jpg)
The death of the data warehouse
![Page 29: SomeSQL at Skyscanner - Scaling in a changing world of databases and hardware](https://reader033.vdocuments.site/reader033/viewer/2022061206/5476d025b4af9f2f628b45cd/html5/thumbnails/29.jpg)
Fluentd
Graphite
Fluentd
Stitchedevents
Stitchedevents
Operationalmetrics
reportingKafka
ErrorsRaw JSON events
ElasticMapReduce
RawEvents
Trigger and view materialization Indexes on the data
A distributed database…
![Page 30: SomeSQL at Skyscanner - Scaling in a changing world of databases and hardware](https://reader033.vdocuments.site/reader033/viewer/2022061206/5476d025b4af9f2f628b45cd/html5/thumbnails/30.jpg)
Some things don’t change
![Page 31: SomeSQL at Skyscanner - Scaling in a changing world of databases and hardware](https://reader033.vdocuments.site/reader033/viewer/2022061206/5476d025b4af9f2f628b45cd/html5/thumbnails/31.jpg)
CC Images courtesy of chaya760 on Flickr
![Page 32: SomeSQL at Skyscanner - Scaling in a changing world of databases and hardware](https://reader033.vdocuments.site/reader033/viewer/2022061206/5476d025b4af9f2f628b45cd/html5/thumbnails/32.jpg)
We still face the same challenges
RAM and Disk i/o concerns
Administration
Security
Data insert and retrieval
Monitoring and alerting
Performance optimization
![Page 33: SomeSQL at Skyscanner - Scaling in a changing world of databases and hardware](https://reader033.vdocuments.site/reader033/viewer/2022061206/5476d025b4af9f2f628b45cd/html5/thumbnails/33.jpg)
The report of my death was an exaggeration
![Page 34: SomeSQL at Skyscanner - Scaling in a changing world of databases and hardware](https://reader033.vdocuments.site/reader033/viewer/2022061206/5476d025b4af9f2f628b45cd/html5/thumbnails/34.jpg)
Elastic Search
![Page 35: SomeSQL at Skyscanner - Scaling in a changing world of databases and hardware](https://reader033.vdocuments.site/reader033/viewer/2022061206/5476d025b4af9f2f628b45cd/html5/thumbnails/35.jpg)
NoSQL
![Page 36: SomeSQL at Skyscanner - Scaling in a changing world of databases and hardware](https://reader033.vdocuments.site/reader033/viewer/2022061206/5476d025b4af9f2f628b45cd/html5/thumbnails/36.jpg)
Microsoft SQL Server
![Page 37: SomeSQL at Skyscanner - Scaling in a changing world of databases and hardware](https://reader033.vdocuments.site/reader033/viewer/2022061206/5476d025b4af9f2f628b45cd/html5/thumbnails/37.jpg)
Relational Vs NoSQL
![Page 38: SomeSQL at Skyscanner - Scaling in a changing world of databases and hardware](https://reader033.vdocuments.site/reader033/viewer/2022061206/5476d025b4af9f2f628b45cd/html5/thumbnails/38.jpg)
EdinburghQuartermile One15 Lauriston PlaceEdinburgh EH3 9EN
Glasgow5th floor, 151-155 St Vincent St, Glasgow G2 5NW
SingaporeNo. 08-01&04 & 09-048th floor, Robinson Point, 39 Robinson Rd, Singapore
BeijingLevel 19, Tower E2, Oriental Plaza, No. 1 East Chang An Avenue, Dong Cheng District, Beijing 100738
Miami1395 Brickell Ave, Suite 900, Miami, Florida 33131
BarcelonaTorre NN, Calle Tarragona, 157, 4a Planta, Barcelona, 08014
thank you