devops oxford- devops + bigdata @ realtime
TRANSCRIPT
DevOps + BigData @ RealTime
DevOps+(DevOps in a startup)
DevOps+● Fixing the electrical outage
DevOps+● Fixing the electrical outage● Cycling to Maplin for an extension lead● Dealing with the sewage from the blocked toilet
BigData● Also known as - Data● BigData is writing Doom or Elite for a 32k machine:
BigData● Or just because you can’t quite fit it on a 39” rotating disk:
@RealTime● Can you survive this ?
The Challenge - The 2014 (Football) World Cup
Analyse every Tweet issued during World Cup games in the 2014 competition
The Challenge - The 2014 (Football) World Cup
The Challenge - The 2014 (Football) World Cup
How busy was it expected to be ?
The Challenge - The 2014 (Football) World Cup
How busy was it expected to be ?● Predicted 1.2M Tweets per hour● Average time for our analysis is 100ms - lengthy texts up to 150ms● Our analysis time increases logarithmically with sentence count and sentence length so needed to avoid pathological cases● 150ms x 1.2M = 180k sec = 3000 minutes = between 33 and 50 hours
The Solution - AWS
● Lots of EC2 instances● 3 Availability Zones● ELB
The Solution - Unattended Build
● Cloud Formation● Puppet● Route53● ELB● 2 min
The Challenge - The 2014 (Football) World Cup
How busy was it ?● Predicted peak 1.2M Tweets per hour● Actual peak 2.5M Tweets per hour● Whole tournament avg 17M per day for 30 days● Peak 33M in one day
The Challenge - The 2014 (Football) World Cup
How busy was it ?● Predicted peak 1.2M Tweets per hour● Actual peak 2.5M Tweets per hour● Whole tournament avg 17M per day for 30 days● Peak 33M in one day
The Challenge - The 2014 (Football) World Cup
How busy was it ?● Predicted peak 1.2M Tweets per hour● Actual peak 2.5M Tweets per hour● Whole tournament avg 17M per day for 30 days● Peak 33M in one day
● A Long Time Ago In A Galaxy Far Far Away● Carhire3000 began - we had 6 servers● Building a new server took a week● 8 years later:-
○ 2 acquisitions○ Booking.com○ $996 million○ 600+ servers○ DevOps team of 5
● Booking.com methodology - Unattended build in 5 minutes● Control Room - Monitoring health of website
● Control Room - Columnar Databases
● Columnar Databases● Fire & Forget UDP events● What do you monitor ?● Experiments & Z-score● DB Queries < 2 seconds● Hot event data in MySQL - Hands Off !● 5 minutes later moved to columnar storage database