scale like an ant, distribute the workload - dpc, amsterdam, 2011
DESCRIPTION
Many services / applications now a day are ill equipped with handling a sudden rush of popularity, as is often the case on the internet now a days, to a point where the services either become unavailable or unbearably slow.By taking a chapter from the ant colonies in the wild, where their strength lies in their numbers and the fact that everyone works together towards the same goal, we can apply the same principle to our service by using systems such as- gearman- memcache- daemons- message queues- load balancersand many more, you can achieve greater performance, more redundancy, higher availability and have the ability to scale your services up and down as required easily.During this talk attendees will be lead through the world of distributed systems and scalability, and shown the how, where and what, of how to take the average application and splitting it into smaller more manageable piecesTRANSCRIPT
![Page 1: Scale like an ant, distribute the workload - DPC, Amsterdam, 2011](https://reader033.vdocuments.site/reader033/viewer/2022042814/554aba15b4c905ec6c8b46cd/html5/thumbnails/1.jpg)
Distribute the workload
Helgi Þormar ÞorbjörnssonDutch PHP Conference, Amsterdam, 19th May 2011
Thursday, 19 May 2011
![Page 2: Scale like an ant, distribute the workload - DPC, Amsterdam, 2011](https://reader033.vdocuments.site/reader033/viewer/2022042814/554aba15b4c905ec6c8b46cd/html5/thumbnails/2.jpg)
Who am I?
Thursday, 19 May 2011
![Page 3: Scale like an ant, distribute the workload - DPC, Amsterdam, 2011](https://reader033.vdocuments.site/reader033/viewer/2022042814/554aba15b4c905ec6c8b46cd/html5/thumbnails/3.jpg)
Helgi
Thursday, 19 May 2011
![Page 4: Scale like an ant, distribute the workload - DPC, Amsterdam, 2011](https://reader033.vdocuments.site/reader033/viewer/2022042814/554aba15b4c905ec6c8b46cd/html5/thumbnails/4.jpg)
VP of Engineering at Orchestra.io
Helgi
Thursday, 19 May 2011
![Page 5: Scale like an ant, distribute the workload - DPC, Amsterdam, 2011](https://reader033.vdocuments.site/reader033/viewer/2022042814/554aba15b4c905ec6c8b46cd/html5/thumbnails/5.jpg)
VP of Engineering at Orchestra.io
Developer at PEAR
Helgi
Thursday, 19 May 2011
![Page 6: Scale like an ant, distribute the workload - DPC, Amsterdam, 2011](https://reader033.vdocuments.site/reader033/viewer/2022042814/554aba15b4c905ec6c8b46cd/html5/thumbnails/6.jpg)
VP of Engineering at Orchestra.io
Developer at PEAR
From Iceland
Helgi
Thursday, 19 May 2011
![Page 7: Scale like an ant, distribute the workload - DPC, Amsterdam, 2011](https://reader033.vdocuments.site/reader033/viewer/2022042814/554aba15b4c905ec6c8b46cd/html5/thumbnails/7.jpg)
VP of Engineering at Orchestra.io
Developer at PEAR
From Iceland
@h on Twitter
Helgi
Thursday, 19 May 2011
![Page 8: Scale like an ant, distribute the workload - DPC, Amsterdam, 2011](https://reader033.vdocuments.site/reader033/viewer/2022042814/554aba15b4c905ec6c8b46cd/html5/thumbnails/8.jpg)
Why Distribute?
Thursday, 19 May 2011
![Page 9: Scale like an ant, distribute the workload - DPC, Amsterdam, 2011](https://reader033.vdocuments.site/reader033/viewer/2022042814/554aba15b4c905ec6c8b46cd/html5/thumbnails/9.jpg)
Why Distribute?
Thursday, 19 May 2011
![Page 10: Scale like an ant, distribute the workload - DPC, Amsterdam, 2011](https://reader033.vdocuments.site/reader033/viewer/2022042814/554aba15b4c905ec6c8b46cd/html5/thumbnails/10.jpg)
Why Distribute?
Budget
Thursday, 19 May 2011
![Page 11: Scale like an ant, distribute the workload - DPC, Amsterdam, 2011](https://reader033.vdocuments.site/reader033/viewer/2022042814/554aba15b4c905ec6c8b46cd/html5/thumbnails/11.jpg)
Why Distribute?
Budget
Efficiency
Thursday, 19 May 2011
![Page 12: Scale like an ant, distribute the workload - DPC, Amsterdam, 2011](https://reader033.vdocuments.site/reader033/viewer/2022042814/554aba15b4c905ec6c8b46cd/html5/thumbnails/12.jpg)
Why Distribute?
Budget
Efficiency
Perception
Thursday, 19 May 2011
![Page 13: Scale like an ant, distribute the workload - DPC, Amsterdam, 2011](https://reader033.vdocuments.site/reader033/viewer/2022042814/554aba15b4c905ec6c8b46cd/html5/thumbnails/13.jpg)
Budget
Thursday, 19 May 2011
![Page 14: Scale like an ant, distribute the workload - DPC, Amsterdam, 2011](https://reader033.vdocuments.site/reader033/viewer/2022042814/554aba15b4c905ec6c8b46cd/html5/thumbnails/14.jpg)
Budget
Spend wisely
Thursday, 19 May 2011
![Page 15: Scale like an ant, distribute the workload - DPC, Amsterdam, 2011](https://reader033.vdocuments.site/reader033/viewer/2022042814/554aba15b4c905ec6c8b46cd/html5/thumbnails/15.jpg)
Budget
Spend wisely
Commodity servers
Thursday, 19 May 2011
![Page 16: Scale like an ant, distribute the workload - DPC, Amsterdam, 2011](https://reader033.vdocuments.site/reader033/viewer/2022042814/554aba15b4c905ec6c8b46cd/html5/thumbnails/16.jpg)
Budget
Spend wisely
Commodity servers
Low overhead, high yield
Thursday, 19 May 2011
![Page 17: Scale like an ant, distribute the workload - DPC, Amsterdam, 2011](https://reader033.vdocuments.site/reader033/viewer/2022042814/554aba15b4c905ec6c8b46cd/html5/thumbnails/17.jpg)
Budget
Spend wisely
Commodity servers
Low overhead, high yield
Cloud Computing (EC2)
Thursday, 19 May 2011
![Page 18: Scale like an ant, distribute the workload - DPC, Amsterdam, 2011](https://reader033.vdocuments.site/reader033/viewer/2022042814/554aba15b4c905ec6c8b46cd/html5/thumbnails/18.jpg)
Efficiency
10 small servers > 1 big
Thursday, 19 May 2011
![Page 19: Scale like an ant, distribute the workload - DPC, Amsterdam, 2011](https://reader033.vdocuments.site/reader033/viewer/2022042814/554aba15b4c905ec6c8b46cd/html5/thumbnails/19.jpg)
Venue Security
Thursday, 19 May 2011
![Page 20: Scale like an ant, distribute the workload - DPC, Amsterdam, 2011](https://reader033.vdocuments.site/reader033/viewer/2022042814/554aba15b4c905ec6c8b46cd/html5/thumbnails/20.jpg)
1000 people can exit quicker through 10 small doors than 1 big
Thursday, 19 May 2011
![Page 21: Scale like an ant, distribute the workload - DPC, Amsterdam, 2011](https://reader033.vdocuments.site/reader033/viewer/2022042814/554aba15b4c905ec6c8b46cd/html5/thumbnails/21.jpg)
1000 people can exit quicker
through 10 small doors than 1 big
Thursday, 19 May 2011
![Page 22: Scale like an ant, distribute the workload - DPC, Amsterdam, 2011](https://reader033.vdocuments.site/reader033/viewer/2022042814/554aba15b4c905ec6c8b46cd/html5/thumbnails/22.jpg)
1000 people can exit quicker
through 10 small doors than 1 big
Thursday, 19 May 2011
![Page 23: Scale like an ant, distribute the workload - DPC, Amsterdam, 2011](https://reader033.vdocuments.site/reader033/viewer/2022042814/554aba15b4c905ec6c8b46cd/html5/thumbnails/23.jpg)
1000 people can exit quicker through 10 small doors than 1 big
Thursday, 19 May 2011
![Page 24: Scale like an ant, distribute the workload - DPC, Amsterdam, 2011](https://reader033.vdocuments.site/reader033/viewer/2022042814/554aba15b4c905ec6c8b46cd/html5/thumbnails/24.jpg)
Thursday, 19 May 2011
![Page 25: Scale like an ant, distribute the workload - DPC, Amsterdam, 2011](https://reader033.vdocuments.site/reader033/viewer/2022042814/554aba15b4c905ec6c8b46cd/html5/thumbnails/25.jpg)
Thursday, 19 May 2011
![Page 26: Scale like an ant, distribute the workload - DPC, Amsterdam, 2011](https://reader033.vdocuments.site/reader033/viewer/2022042814/554aba15b4c905ec6c8b46cd/html5/thumbnails/26.jpg)
Thursday, 19 May 2011
![Page 27: Scale like an ant, distribute the workload - DPC, Amsterdam, 2011](https://reader033.vdocuments.site/reader033/viewer/2022042814/554aba15b4c905ec6c8b46cd/html5/thumbnails/27.jpg)
Thursday, 19 May 2011
![Page 28: Scale like an ant, distribute the workload - DPC, Amsterdam, 2011](https://reader033.vdocuments.site/reader033/viewer/2022042814/554aba15b4c905ec6c8b46cd/html5/thumbnails/28.jpg)
Perception
Thursday, 19 May 2011
![Page 29: Scale like an ant, distribute the workload - DPC, Amsterdam, 2011](https://reader033.vdocuments.site/reader033/viewer/2022042814/554aba15b4c905ec6c8b46cd/html5/thumbnails/29.jpg)
Perception
Defer intensive processes
Thursday, 19 May 2011
![Page 30: Scale like an ant, distribute the workload - DPC, Amsterdam, 2011](https://reader033.vdocuments.site/reader033/viewer/2022042814/554aba15b4c905ec6c8b46cd/html5/thumbnails/30.jpg)
Perception
Defer intensive processes
Give instant feedback
Thursday, 19 May 2011
![Page 31: Scale like an ant, distribute the workload - DPC, Amsterdam, 2011](https://reader033.vdocuments.site/reader033/viewer/2022042814/554aba15b4c905ec6c8b46cd/html5/thumbnails/31.jpg)
Perception
Defer intensive processes
Give instant feedback
Users keep on browsing
Thursday, 19 May 2011
![Page 32: Scale like an ant, distribute the workload - DPC, Amsterdam, 2011](https://reader033.vdocuments.site/reader033/viewer/2022042814/554aba15b4c905ec6c8b46cd/html5/thumbnails/32.jpg)
Perception
Defer intensive processes
Give instant feedback
Users keep on browsing
Thursday, 19 May 2011
![Page 33: Scale like an ant, distribute the workload - DPC, Amsterdam, 2011](https://reader033.vdocuments.site/reader033/viewer/2022042814/554aba15b4c905ec6c8b46cd/html5/thumbnails/33.jpg)
“It all depends on how we look at things, and not how
they are in themselves.”
- Carl G. Jung
Thursday, 19 May 2011
![Page 34: Scale like an ant, distribute the workload - DPC, Amsterdam, 2011](https://reader033.vdocuments.site/reader033/viewer/2022042814/554aba15b4c905ec6c8b46cd/html5/thumbnails/34.jpg)
Thursday, 19 May 2011
![Page 35: Scale like an ant, distribute the workload - DPC, Amsterdam, 2011](https://reader033.vdocuments.site/reader033/viewer/2022042814/554aba15b4c905ec6c8b46cd/html5/thumbnails/35.jpg)
Chapter from Nature
Thursday, 19 May 2011
![Page 36: Scale like an ant, distribute the workload - DPC, Amsterdam, 2011](https://reader033.vdocuments.site/reader033/viewer/2022042814/554aba15b4c905ec6c8b46cd/html5/thumbnails/36.jpg)
Ant Colonies
Thursday, 19 May 2011
![Page 37: Scale like an ant, distribute the workload - DPC, Amsterdam, 2011](https://reader033.vdocuments.site/reader033/viewer/2022042814/554aba15b4c905ec6c8b46cd/html5/thumbnails/37.jpg)
Algorithms
Thursday, 19 May 2011
![Page 38: Scale like an ant, distribute the workload - DPC, Amsterdam, 2011](https://reader033.vdocuments.site/reader033/viewer/2022042814/554aba15b4c905ec6c8b46cd/html5/thumbnails/38.jpg)
Algorithms
Scheduling
Thursday, 19 May 2011
![Page 39: Scale like an ant, distribute the workload - DPC, Amsterdam, 2011](https://reader033.vdocuments.site/reader033/viewer/2022042814/554aba15b4c905ec6c8b46cd/html5/thumbnails/39.jpg)
Algorithms
Scheduling
Vehicle Routing
Thursday, 19 May 2011
![Page 40: Scale like an ant, distribute the workload - DPC, Amsterdam, 2011](https://reader033.vdocuments.site/reader033/viewer/2022042814/554aba15b4c905ec6c8b46cd/html5/thumbnails/40.jpg)
Algorithms
Scheduling
Vehicle Routing
Assignment
Thursday, 19 May 2011
![Page 41: Scale like an ant, distribute the workload - DPC, Amsterdam, 2011](https://reader033.vdocuments.site/reader033/viewer/2022042814/554aba15b4c905ec6c8b46cd/html5/thumbnails/41.jpg)
Algorithms
Scheduling
Vehicle Routing
Assignment
Sets
Thursday, 19 May 2011
![Page 42: Scale like an ant, distribute the workload - DPC, Amsterdam, 2011](https://reader033.vdocuments.site/reader033/viewer/2022042814/554aba15b4c905ec6c8b46cd/html5/thumbnails/42.jpg)
Algorithms
Scheduling
Vehicle Routing
Assignment
Sets
Other
Thursday, 19 May 2011
![Page 43: Scale like an ant, distribute the workload - DPC, Amsterdam, 2011](https://reader033.vdocuments.site/reader033/viewer/2022042814/554aba15b4c905ec6c8b46cd/html5/thumbnails/43.jpg)
Algorithms
Scheduling
Vehicle Routing
Assignment
Sets
Other
Thursday, 19 May 2011
![Page 44: Scale like an ant, distribute the workload - DPC, Amsterdam, 2011](https://reader033.vdocuments.site/reader033/viewer/2022042814/554aba15b4c905ec6c8b46cd/html5/thumbnails/44.jpg)
How do ants fit?
Thursday, 19 May 2011
![Page 45: Scale like an ant, distribute the workload - DPC, Amsterdam, 2011](https://reader033.vdocuments.site/reader033/viewer/2022042814/554aba15b4c905ec6c8b46cd/html5/thumbnails/45.jpg)
How do ants fit?
Strength in numbers
Thursday, 19 May 2011
![Page 46: Scale like an ant, distribute the workload - DPC, Amsterdam, 2011](https://reader033.vdocuments.site/reader033/viewer/2022042814/554aba15b4c905ec6c8b46cd/html5/thumbnails/46.jpg)
How do ants fit?
Strength in numbers
Work together
Thursday, 19 May 2011
![Page 47: Scale like an ant, distribute the workload - DPC, Amsterdam, 2011](https://reader033.vdocuments.site/reader033/viewer/2022042814/554aba15b4c905ec6c8b46cd/html5/thumbnails/47.jpg)
How do ants fit?
Strength in numbers
Work together
Size benefits them
Thursday, 19 May 2011
![Page 48: Scale like an ant, distribute the workload - DPC, Amsterdam, 2011](https://reader033.vdocuments.site/reader033/viewer/2022042814/554aba15b4c905ec6c8b46cd/html5/thumbnails/48.jpg)
Teamwork
When faced with a problem they will solve the problem as one.
Thursday, 19 May 2011
![Page 49: Scale like an ant, distribute the workload - DPC, Amsterdam, 2011](https://reader033.vdocuments.site/reader033/viewer/2022042814/554aba15b4c905ec6c8b46cd/html5/thumbnails/49.jpg)
Thursday, 19 May 2011
![Page 50: Scale like an ant, distribute the workload - DPC, Amsterdam, 2011](https://reader033.vdocuments.site/reader033/viewer/2022042814/554aba15b4c905ec6c8b46cd/html5/thumbnails/50.jpg)
Thursday, 19 May 2011
![Page 51: Scale like an ant, distribute the workload - DPC, Amsterdam, 2011](https://reader033.vdocuments.site/reader033/viewer/2022042814/554aba15b4c905ec6c8b46cd/html5/thumbnails/51.jpg)
What if they were bigger?
Thursday, 19 May 2011
![Page 52: Scale like an ant, distribute the workload - DPC, Amsterdam, 2011](https://reader033.vdocuments.site/reader033/viewer/2022042814/554aba15b4c905ec6c8b46cd/html5/thumbnails/52.jpg)
Types of Ants
Thursday, 19 May 2011
![Page 53: Scale like an ant, distribute the workload - DPC, Amsterdam, 2011](https://reader033.vdocuments.site/reader033/viewer/2022042814/554aba15b4c905ec6c8b46cd/html5/thumbnails/53.jpg)
Types of Ants
Military
Thursday, 19 May 2011
![Page 54: Scale like an ant, distribute the workload - DPC, Amsterdam, 2011](https://reader033.vdocuments.site/reader033/viewer/2022042814/554aba15b4c905ec6c8b46cd/html5/thumbnails/54.jpg)
Types of Ants
Military
Maids
Thursday, 19 May 2011
![Page 55: Scale like an ant, distribute the workload - DPC, Amsterdam, 2011](https://reader033.vdocuments.site/reader033/viewer/2022042814/554aba15b4c905ec6c8b46cd/html5/thumbnails/55.jpg)
Types of Ants
Military
Maids
Tunnel diggers
Thursday, 19 May 2011
![Page 56: Scale like an ant, distribute the workload - DPC, Amsterdam, 2011](https://reader033.vdocuments.site/reader033/viewer/2022042814/554aba15b4c905ec6c8b46cd/html5/thumbnails/56.jpg)
Types of Ants
Military
Maids
Tunnel diggers
Food gatherers
Thursday, 19 May 2011
![Page 57: Scale like an ant, distribute the workload - DPC, Amsterdam, 2011](https://reader033.vdocuments.site/reader033/viewer/2022042814/554aba15b4c905ec6c8b46cd/html5/thumbnails/57.jpg)
How does this map to my application?
Thursday, 19 May 2011
![Page 58: Scale like an ant, distribute the workload - DPC, Amsterdam, 2011](https://reader033.vdocuments.site/reader033/viewer/2022042814/554aba15b4c905ec6c8b46cd/html5/thumbnails/58.jpg)
Thursday, 19 May 2011
![Page 59: Scale like an ant, distribute the workload - DPC, Amsterdam, 2011](https://reader033.vdocuments.site/reader033/viewer/2022042814/554aba15b4c905ec6c8b46cd/html5/thumbnails/59.jpg)
Colony = Application
Thursday, 19 May 2011
![Page 60: Scale like an ant, distribute the workload - DPC, Amsterdam, 2011](https://reader033.vdocuments.site/reader033/viewer/2022042814/554aba15b4c905ec6c8b46cd/html5/thumbnails/60.jpg)
Colony = Application Ants = Components
Thursday, 19 May 2011
![Page 61: Scale like an ant, distribute the workload - DPC, Amsterdam, 2011](https://reader033.vdocuments.site/reader033/viewer/2022042814/554aba15b4c905ec6c8b46cd/html5/thumbnails/61.jpg)
Colony = Application Ants = Components
Ants do many different types of work to keep their colony running
Thursday, 19 May 2011
![Page 62: Scale like an ant, distribute the workload - DPC, Amsterdam, 2011](https://reader033.vdocuments.site/reader033/viewer/2022042814/554aba15b4c905ec6c8b46cd/html5/thumbnails/62.jpg)
Architect for Distribution
Thursday, 19 May 2011
![Page 63: Scale like an ant, distribute the workload - DPC, Amsterdam, 2011](https://reader033.vdocuments.site/reader033/viewer/2022042814/554aba15b4c905ec6c8b46cd/html5/thumbnails/63.jpg)
Characteristics
Thursday, 19 May 2011
![Page 64: Scale like an ant, distribute the workload - DPC, Amsterdam, 2011](https://reader033.vdocuments.site/reader033/viewer/2022042814/554aba15b4c905ec6c8b46cd/html5/thumbnails/64.jpg)
Characteristics
Decoupling
Thursday, 19 May 2011
![Page 65: Scale like an ant, distribute the workload - DPC, Amsterdam, 2011](https://reader033.vdocuments.site/reader033/viewer/2022042814/554aba15b4c905ec6c8b46cd/html5/thumbnails/65.jpg)
Characteristics
Decoupling
Elasticity
Thursday, 19 May 2011
![Page 66: Scale like an ant, distribute the workload - DPC, Amsterdam, 2011](https://reader033.vdocuments.site/reader033/viewer/2022042814/554aba15b4c905ec6c8b46cd/html5/thumbnails/66.jpg)
Characteristics
Decoupling
Elasticity
High Availability
Thursday, 19 May 2011
![Page 67: Scale like an ant, distribute the workload - DPC, Amsterdam, 2011](https://reader033.vdocuments.site/reader033/viewer/2022042814/554aba15b4c905ec6c8b46cd/html5/thumbnails/67.jpg)
Characteristics
Decoupling
Elasticity
High Availability
Concurrency
Thursday, 19 May 2011
![Page 68: Scale like an ant, distribute the workload - DPC, Amsterdam, 2011](https://reader033.vdocuments.site/reader033/viewer/2022042814/554aba15b4c905ec6c8b46cd/html5/thumbnails/68.jpg)
Decoupling
Thursday, 19 May 2011
![Page 69: Scale like an ant, distribute the workload - DPC, Amsterdam, 2011](https://reader033.vdocuments.site/reader033/viewer/2022042814/554aba15b4c905ec6c8b46cd/html5/thumbnails/69.jpg)
Application
DB API
Cache FE
Thursday, 19 May 2011
![Page 70: Scale like an ant, distribute the workload - DPC, Amsterdam, 2011](https://reader033.vdocuments.site/reader033/viewer/2022042814/554aba15b4c905ec6c8b46cd/html5/thumbnails/70.jpg)
Application
DB API
Cache FE
Thursday, 19 May 2011
![Page 71: Scale like an ant, distribute the workload - DPC, Amsterdam, 2011](https://reader033.vdocuments.site/reader033/viewer/2022042814/554aba15b4c905ec6c8b46cd/html5/thumbnails/71.jpg)
ApplicationDB API
Cache FE
Thursday, 19 May 2011
![Page 72: Scale like an ant, distribute the workload - DPC, Amsterdam, 2011](https://reader033.vdocuments.site/reader033/viewer/2022042814/554aba15b4c905ec6c8b46cd/html5/thumbnails/72.jpg)
ApplicationDB API
Cache FE
Cache
Thursday, 19 May 2011
![Page 73: Scale like an ant, distribute the workload - DPC, Amsterdam, 2011](https://reader033.vdocuments.site/reader033/viewer/2022042814/554aba15b4c905ec6c8b46cd/html5/thumbnails/73.jpg)
ApplicationDB API
Cache FE
Cache
API
Thursday, 19 May 2011
![Page 74: Scale like an ant, distribute the workload - DPC, Amsterdam, 2011](https://reader033.vdocuments.site/reader033/viewer/2022042814/554aba15b4c905ec6c8b46cd/html5/thumbnails/74.jpg)
ApplicationDB API
Cache FE
Cache
API
API
Thursday, 19 May 2011
![Page 75: Scale like an ant, distribute the workload - DPC, Amsterdam, 2011](https://reader033.vdocuments.site/reader033/viewer/2022042814/554aba15b4c905ec6c8b46cd/html5/thumbnails/75.jpg)
Elasticity
Thursday, 19 May 2011
![Page 76: Scale like an ant, distribute the workload - DPC, Amsterdam, 2011](https://reader033.vdocuments.site/reader033/viewer/2022042814/554aba15b4c905ec6c8b46cd/html5/thumbnails/76.jpg)
Cloud Computing
Thursday, 19 May 2011
![Page 77: Scale like an ant, distribute the workload - DPC, Amsterdam, 2011](https://reader033.vdocuments.site/reader033/viewer/2022042814/554aba15b4c905ec6c8b46cd/html5/thumbnails/77.jpg)
Load Balancing
Thursday, 19 May 2011
![Page 78: Scale like an ant, distribute the workload - DPC, Amsterdam, 2011](https://reader033.vdocuments.site/reader033/viewer/2022042814/554aba15b4c905ec6c8b46cd/html5/thumbnails/78.jpg)
HA Proxy
Nginx
My Favourite
Thursday, 19 May 2011
![Page 79: Scale like an ant, distribute the workload - DPC, Amsterdam, 2011](https://reader033.vdocuments.site/reader033/viewer/2022042814/554aba15b4c905ec6c8b46cd/html5/thumbnails/79.jpg)
Monitoring
Thursday, 19 May 2011
![Page 80: Scale like an ant, distribute the workload - DPC, Amsterdam, 2011](https://reader033.vdocuments.site/reader033/viewer/2022042814/554aba15b4c905ec6c8b46cd/html5/thumbnails/80.jpg)
When do I need more servers?
Thursday, 19 May 2011
![Page 81: Scale like an ant, distribute the workload - DPC, Amsterdam, 2011](https://reader033.vdocuments.site/reader033/viewer/2022042814/554aba15b4c905ec6c8b46cd/html5/thumbnails/81.jpg)
Needs to be around from the start!
Thursday, 19 May 2011
![Page 82: Scale like an ant, distribute the workload - DPC, Amsterdam, 2011](https://reader033.vdocuments.site/reader033/viewer/2022042814/554aba15b4c905ec6c8b46cd/html5/thumbnails/82.jpg)
Keep records
Thursday, 19 May 2011
![Page 83: Scale like an ant, distribute the workload - DPC, Amsterdam, 2011](https://reader033.vdocuments.site/reader033/viewer/2022042814/554aba15b4c905ec6c8b46cd/html5/thumbnails/83.jpg)
Spot trends
Thursday, 19 May 2011
![Page 84: Scale like an ant, distribute the workload - DPC, Amsterdam, 2011](https://reader033.vdocuments.site/reader033/viewer/2022042814/554aba15b4c905ec6c8b46cd/html5/thumbnails/84.jpg)
Different types
Thursday, 19 May 2011
![Page 85: Scale like an ant, distribute the workload - DPC, Amsterdam, 2011](https://reader033.vdocuments.site/reader033/viewer/2022042814/554aba15b4c905ec6c8b46cd/html5/thumbnails/85.jpg)
Different types
Hardware Performance
Thursday, 19 May 2011
![Page 86: Scale like an ant, distribute the workload - DPC, Amsterdam, 2011](https://reader033.vdocuments.site/reader033/viewer/2022042814/554aba15b4c905ec6c8b46cd/html5/thumbnails/86.jpg)
Different types
Hardware Performance
Software Performance
Thursday, 19 May 2011
![Page 87: Scale like an ant, distribute the workload - DPC, Amsterdam, 2011](https://reader033.vdocuments.site/reader033/viewer/2022042814/554aba15b4c905ec6c8b46cd/html5/thumbnails/87.jpg)
Different types
Hardware Performance
Software Performance
Availability
Thursday, 19 May 2011
![Page 88: Scale like an ant, distribute the workload - DPC, Amsterdam, 2011](https://reader033.vdocuments.site/reader033/viewer/2022042814/554aba15b4c905ec6c8b46cd/html5/thumbnails/88.jpg)
Different types
Hardware Performance
Software Performance
Availability
Resourcing
Thursday, 19 May 2011
![Page 89: Scale like an ant, distribute the workload - DPC, Amsterdam, 2011](https://reader033.vdocuments.site/reader033/viewer/2022042814/554aba15b4c905ec6c8b46cd/html5/thumbnails/89.jpg)
Different types
Hardware Performance
Software Performance
Availability
Resourcing
Thursday, 19 May 2011
![Page 90: Scale like an ant, distribute the workload - DPC, Amsterdam, 2011](https://reader033.vdocuments.site/reader033/viewer/2022042814/554aba15b4c905ec6c8b46cd/html5/thumbnails/90.jpg)
Applications
Thursday, 19 May 2011
![Page 91: Scale like an ant, distribute the workload - DPC, Amsterdam, 2011](https://reader033.vdocuments.site/reader033/viewer/2022042814/554aba15b4c905ec6c8b46cd/html5/thumbnails/91.jpg)
ApplicationsNew Relic
Thursday, 19 May 2011
![Page 92: Scale like an ant, distribute the workload - DPC, Amsterdam, 2011](https://reader033.vdocuments.site/reader033/viewer/2022042814/554aba15b4c905ec6c8b46cd/html5/thumbnails/92.jpg)
ApplicationsNew Relic
CloudKick
Thursday, 19 May 2011
![Page 93: Scale like an ant, distribute the workload - DPC, Amsterdam, 2011](https://reader033.vdocuments.site/reader033/viewer/2022042814/554aba15b4c905ec6c8b46cd/html5/thumbnails/93.jpg)
ApplicationsNew Relic
CloudKick
ScoutApp
Thursday, 19 May 2011
![Page 94: Scale like an ant, distribute the workload - DPC, Amsterdam, 2011](https://reader033.vdocuments.site/reader033/viewer/2022042814/554aba15b4c905ec6c8b46cd/html5/thumbnails/94.jpg)
ApplicationsNew Relic
CloudKick
ScoutApp
Nagios
Thursday, 19 May 2011
![Page 95: Scale like an ant, distribute the workload - DPC, Amsterdam, 2011](https://reader033.vdocuments.site/reader033/viewer/2022042814/554aba15b4c905ec6c8b46cd/html5/thumbnails/95.jpg)
ApplicationsNew Relic
CloudKick
ScoutApp
Nagios
Cacti
Thursday, 19 May 2011
![Page 96: Scale like an ant, distribute the workload - DPC, Amsterdam, 2011](https://reader033.vdocuments.site/reader033/viewer/2022042814/554aba15b4c905ec6c8b46cd/html5/thumbnails/96.jpg)
ApplicationsNew Relic
CloudKick
ScoutApp
Nagios
Cacti
Circonus
Thursday, 19 May 2011
![Page 97: Scale like an ant, distribute the workload - DPC, Amsterdam, 2011](https://reader033.vdocuments.site/reader033/viewer/2022042814/554aba15b4c905ec6c8b46cd/html5/thumbnails/97.jpg)
Automation
Thursday, 19 May 2011
![Page 98: Scale like an ant, distribute the workload - DPC, Amsterdam, 2011](https://reader033.vdocuments.site/reader033/viewer/2022042814/554aba15b4c905ec6c8b46cd/html5/thumbnails/98.jpg)
Want to sleep easy at night?
Thursday, 19 May 2011
![Page 99: Scale like an ant, distribute the workload - DPC, Amsterdam, 2011](https://reader033.vdocuments.site/reader033/viewer/2022042814/554aba15b4c905ec6c8b46cd/html5/thumbnails/99.jpg)
Want to sleep easy at night?
Go out partying without worrying about getting a phone call?
Thursday, 19 May 2011
![Page 100: Scale like an ant, distribute the workload - DPC, Amsterdam, 2011](https://reader033.vdocuments.site/reader033/viewer/2022042814/554aba15b4c905ec6c8b46cd/html5/thumbnails/100.jpg)
Plug into your monitoring
Thursday, 19 May 2011
![Page 101: Scale like an ant, distribute the workload - DPC, Amsterdam, 2011](https://reader033.vdocuments.site/reader033/viewer/2022042814/554aba15b4c905ec6c8b46cd/html5/thumbnails/101.jpg)
Bringing together Monitoring and Elastic behaviour into one
beautiful whole!
Thursday, 19 May 2011
![Page 102: Scale like an ant, distribute the workload - DPC, Amsterdam, 2011](https://reader033.vdocuments.site/reader033/viewer/2022042814/554aba15b4c905ec6c8b46cd/html5/thumbnails/102.jpg)
Add some intelligence to add / remove servers as needed based
on current information.
Thursday, 19 May 2011
![Page 103: Scale like an ant, distribute the workload - DPC, Amsterdam, 2011](https://reader033.vdocuments.site/reader033/viewer/2022042814/554aba15b4c905ec6c8b46cd/html5/thumbnails/103.jpg)
This is why good monitoring is essential or this wouldn’t be
possible
Thursday, 19 May 2011
![Page 104: Scale like an ant, distribute the workload - DPC, Amsterdam, 2011](https://reader033.vdocuments.site/reader033/viewer/2022042814/554aba15b4c905ec6c8b46cd/html5/thumbnails/104.jpg)
Just make sure it doesn’t turn into...
Thursday, 19 May 2011
![Page 105: Scale like an ant, distribute the workload - DPC, Amsterdam, 2011](https://reader033.vdocuments.site/reader033/viewer/2022042814/554aba15b4c905ec6c8b46cd/html5/thumbnails/105.jpg)
Skynet!!Thursday, 19 May 2011
![Page 106: Scale like an ant, distribute the workload - DPC, Amsterdam, 2011](https://reader033.vdocuments.site/reader033/viewer/2022042814/554aba15b4c905ec6c8b46cd/html5/thumbnails/106.jpg)
High Availability
Thursday, 19 May 2011
![Page 107: Scale like an ant, distribute the workload - DPC, Amsterdam, 2011](https://reader033.vdocuments.site/reader033/viewer/2022042814/554aba15b4c905ec6c8b46cd/html5/thumbnails/107.jpg)
Get a highly available and resilient setup by following a few
of those recommendations
Thursday, 19 May 2011
![Page 108: Scale like an ant, distribute the workload - DPC, Amsterdam, 2011](https://reader033.vdocuments.site/reader033/viewer/2022042814/554aba15b4c905ec6c8b46cd/html5/thumbnails/108.jpg)
Remember, even Google has outages
Thursday, 19 May 2011
![Page 109: Scale like an ant, distribute the workload - DPC, Amsterdam, 2011](https://reader033.vdocuments.site/reader033/viewer/2022042814/554aba15b4c905ec6c8b46cd/html5/thumbnails/109.jpg)
Benefits
Thursday, 19 May 2011
![Page 110: Scale like an ant, distribute the workload - DPC, Amsterdam, 2011](https://reader033.vdocuments.site/reader033/viewer/2022042814/554aba15b4c905ec6c8b46cd/html5/thumbnails/110.jpg)
Benefits
Easy management
Thursday, 19 May 2011
![Page 111: Scale like an ant, distribute the workload - DPC, Amsterdam, 2011](https://reader033.vdocuments.site/reader033/viewer/2022042814/554aba15b4c905ec6c8b46cd/html5/thumbnails/111.jpg)
Benefits
Easy management
Ability to stop / start servers quickly
Thursday, 19 May 2011
![Page 112: Scale like an ant, distribute the workload - DPC, Amsterdam, 2011](https://reader033.vdocuments.site/reader033/viewer/2022042814/554aba15b4c905ec6c8b46cd/html5/thumbnails/112.jpg)
Benefits
Easy management
Ability to stop / start servers quickly
Responsibilities are separate
Thursday, 19 May 2011
![Page 113: Scale like an ant, distribute the workload - DPC, Amsterdam, 2011](https://reader033.vdocuments.site/reader033/viewer/2022042814/554aba15b4c905ec6c8b46cd/html5/thumbnails/113.jpg)
Benefits
Easy management
Ability to stop / start servers quickly
Responsibilities are separate
Quickly move to a new cluster
Thursday, 19 May 2011
![Page 114: Scale like an ant, distribute the workload - DPC, Amsterdam, 2011](https://reader033.vdocuments.site/reader033/viewer/2022042814/554aba15b4c905ec6c8b46cd/html5/thumbnails/114.jpg)
Benefits
Easy management
Ability to stop / start servers quickly
Responsibilities are separate
Quickly move to a new cluster
Reduced risk
Thursday, 19 May 2011
![Page 115: Scale like an ant, distribute the workload - DPC, Amsterdam, 2011](https://reader033.vdocuments.site/reader033/viewer/2022042814/554aba15b4c905ec6c8b46cd/html5/thumbnails/115.jpg)
What to avoid
Thursday, 19 May 2011
![Page 116: Scale like an ant, distribute the workload - DPC, Amsterdam, 2011](https://reader033.vdocuments.site/reader033/viewer/2022042814/554aba15b4c905ec6c8b46cd/html5/thumbnails/116.jpg)
Local Sessions
Thursday, 19 May 2011
![Page 117: Scale like an ant, distribute the workload - DPC, Amsterdam, 2011](https://reader033.vdocuments.site/reader033/viewer/2022042814/554aba15b4c905ec6c8b46cd/html5/thumbnails/117.jpg)
Store sessions in DB / Memcache
Solution
Thursday, 19 May 2011
![Page 118: Scale like an ant, distribute the workload - DPC, Amsterdam, 2011](https://reader033.vdocuments.site/reader033/viewer/2022042814/554aba15b4c905ec6c8b46cd/html5/thumbnails/118.jpg)
Local Memory
Thursday, 19 May 2011
![Page 119: Scale like an ant, distribute the workload - DPC, Amsterdam, 2011](https://reader033.vdocuments.site/reader033/viewer/2022042814/554aba15b4c905ec6c8b46cd/html5/thumbnails/119.jpg)
Networked Memcache
Solution
Thursday, 19 May 2011
![Page 120: Scale like an ant, distribute the workload - DPC, Amsterdam, 2011](https://reader033.vdocuments.site/reader033/viewer/2022042814/554aba15b4c905ec6c8b46cd/html5/thumbnails/120.jpg)
Local Files
Thursday, 19 May 2011
![Page 121: Scale like an ant, distribute the workload - DPC, Amsterdam, 2011](https://reader033.vdocuments.site/reader033/viewer/2022042814/554aba15b4c905ec6c8b46cd/html5/thumbnails/121.jpg)
Local Uploads
Thursday, 19 May 2011
![Page 122: Scale like an ant, distribute the workload - DPC, Amsterdam, 2011](https://reader033.vdocuments.site/reader033/viewer/2022042814/554aba15b4c905ec6c8b46cd/html5/thumbnails/122.jpg)
Writing to /tmp
Thursday, 19 May 2011
![Page 123: Scale like an ant, distribute the workload - DPC, Amsterdam, 2011](https://reader033.vdocuments.site/reader033/viewer/2022042814/554aba15b4c905ec6c8b46cd/html5/thumbnails/123.jpg)
Store on S3 or a networked FS
Solution
Thursday, 19 May 2011
![Page 124: Scale like an ant, distribute the workload - DPC, Amsterdam, 2011](https://reader033.vdocuments.site/reader033/viewer/2022042814/554aba15b4c905ec6c8b46cd/html5/thumbnails/124.jpg)
Serve up static files from CDNs
Solution
Thursday, 19 May 2011
![Page 125: Scale like an ant, distribute the workload - DPC, Amsterdam, 2011](https://reader033.vdocuments.site/reader033/viewer/2022042814/554aba15b4c905ec6c8b46cd/html5/thumbnails/125.jpg)
Servers can vanish at any given time
Thursday, 19 May 2011
![Page 126: Scale like an ant, distribute the workload - DPC, Amsterdam, 2011](https://reader033.vdocuments.site/reader033/viewer/2022042814/554aba15b4c905ec6c8b46cd/html5/thumbnails/126.jpg)
Internal APIs
Thursday, 19 May 2011
![Page 127: Scale like an ant, distribute the workload - DPC, Amsterdam, 2011](https://reader033.vdocuments.site/reader033/viewer/2022042814/554aba15b4c905ec6c8b46cd/html5/thumbnails/127.jpg)
Application
S3GFS FS
Internal Storage API
Thursday, 19 May 2011
![Page 128: Scale like an ant, distribute the workload - DPC, Amsterdam, 2011](https://reader033.vdocuments.site/reader033/viewer/2022042814/554aba15b4c905ec6c8b46cd/html5/thumbnails/128.jpg)
Application
MySQLMongo Cache
Internal DB API
Thursday, 19 May 2011
![Page 129: Scale like an ant, distribute the workload - DPC, Amsterdam, 2011](https://reader033.vdocuments.site/reader033/viewer/2022042814/554aba15b4c905ec6c8b46cd/html5/thumbnails/129.jpg)
SOA
Thursday, 19 May 2011
![Page 130: Scale like an ant, distribute the workload - DPC, Amsterdam, 2011](https://reader033.vdocuments.site/reader033/viewer/2022042814/554aba15b4c905ec6c8b46cd/html5/thumbnails/130.jpg)
Service Oriented Architecture
Thursday, 19 May 2011
![Page 131: Scale like an ant, distribute the workload - DPC, Amsterdam, 2011](https://reader033.vdocuments.site/reader033/viewer/2022042814/554aba15b4c905ec6c8b46cd/html5/thumbnails/131.jpg)
Sort of :-)
Thursday, 19 May 2011
![Page 132: Scale like an ant, distribute the workload - DPC, Amsterdam, 2011](https://reader033.vdocuments.site/reader033/viewer/2022042814/554aba15b4c905ec6c8b46cd/html5/thumbnails/132.jpg)
Eventually Consistent
Thursday, 19 May 2011
![Page 133: Scale like an ant, distribute the workload - DPC, Amsterdam, 2011](https://reader033.vdocuments.site/reader033/viewer/2022042814/554aba15b4c905ec6c8b46cd/html5/thumbnails/133.jpg)
CAP Therom
Thursday, 19 May 2011
![Page 134: Scale like an ant, distribute the workload - DPC, Amsterdam, 2011](https://reader033.vdocuments.site/reader033/viewer/2022042814/554aba15b4c905ec6c8b46cd/html5/thumbnails/134.jpg)
Consistency
Availability
Partition Tolerance
Thursday, 19 May 2011
![Page 135: Scale like an ant, distribute the workload - DPC, Amsterdam, 2011](https://reader033.vdocuments.site/reader033/viewer/2022042814/554aba15b4c905ec6c8b46cd/html5/thumbnails/135.jpg)
Consistency
All nodes see the same data at the same time
Thursday, 19 May 2011
![Page 136: Scale like an ant, distribute the workload - DPC, Amsterdam, 2011](https://reader033.vdocuments.site/reader033/viewer/2022042814/554aba15b4c905ec6c8b46cd/html5/thumbnails/136.jpg)
Availability
Node failures do not prevent survivors from continuing to
operate
Thursday, 19 May 2011
![Page 137: Scale like an ant, distribute the workload - DPC, Amsterdam, 2011](https://reader033.vdocuments.site/reader033/viewer/2022042814/554aba15b4c905ec6c8b46cd/html5/thumbnails/137.jpg)
Partition Tolerance
The system continues to operate despite arbitrary message loss
Thursday, 19 May 2011
![Page 138: Scale like an ant, distribute the workload - DPC, Amsterdam, 2011](https://reader033.vdocuments.site/reader033/viewer/2022042814/554aba15b4c905ec6c8b46cd/html5/thumbnails/138.jpg)
Consistency
Availability
Partition Tolerance
Thursday, 19 May 2011
![Page 139: Scale like an ant, distribute the workload - DPC, Amsterdam, 2011](https://reader033.vdocuments.site/reader033/viewer/2022042814/554aba15b4c905ec6c8b46cd/html5/thumbnails/139.jpg)
Queue Systems
Thursday, 19 May 2011
![Page 140: Scale like an ant, distribute the workload - DPC, Amsterdam, 2011](https://reader033.vdocuments.site/reader033/viewer/2022042814/554aba15b4c905ec6c8b46cd/html5/thumbnails/140.jpg)
Good for
Thursday, 19 May 2011
![Page 141: Scale like an ant, distribute the workload - DPC, Amsterdam, 2011](https://reader033.vdocuments.site/reader033/viewer/2022042814/554aba15b4c905ec6c8b46cd/html5/thumbnails/141.jpg)
Good forImage Processing
Thursday, 19 May 2011
![Page 142: Scale like an ant, distribute the workload - DPC, Amsterdam, 2011](https://reader033.vdocuments.site/reader033/viewer/2022042814/554aba15b4c905ec6c8b46cd/html5/thumbnails/142.jpg)
Good forImage Processing
Distributed Logs
Thursday, 19 May 2011
![Page 143: Scale like an ant, distribute the workload - DPC, Amsterdam, 2011](https://reader033.vdocuments.site/reader033/viewer/2022042814/554aba15b4c905ec6c8b46cd/html5/thumbnails/143.jpg)
Good forImage Processing
Distributed Logs
Data Mining
Thursday, 19 May 2011
![Page 144: Scale like an ant, distribute the workload - DPC, Amsterdam, 2011](https://reader033.vdocuments.site/reader033/viewer/2022042814/554aba15b4c905ec6c8b46cd/html5/thumbnails/144.jpg)
Good forImage Processing
Distributed Logs
Data Mining
Mass Emails
Thursday, 19 May 2011
![Page 145: Scale like an ant, distribute the workload - DPC, Amsterdam, 2011](https://reader033.vdocuments.site/reader033/viewer/2022042814/554aba15b4c905ec6c8b46cd/html5/thumbnails/145.jpg)
Good forImage Processing
Distributed Logs
Data Mining
Mass Emails
Intensive transformation
Thursday, 19 May 2011
![Page 146: Scale like an ant, distribute the workload - DPC, Amsterdam, 2011](https://reader033.vdocuments.site/reader033/viewer/2022042814/554aba15b4c905ec6c8b46cd/html5/thumbnails/146.jpg)
Good forImage Processing
Distributed Logs
Data Mining
Mass Emails
Intensive transformation
Search
Thursday, 19 May 2011
![Page 147: Scale like an ant, distribute the workload - DPC, Amsterdam, 2011](https://reader033.vdocuments.site/reader033/viewer/2022042814/554aba15b4c905ec6c8b46cd/html5/thumbnails/147.jpg)
Common Tools
Thursday, 19 May 2011
![Page 148: Scale like an ant, distribute the workload - DPC, Amsterdam, 2011](https://reader033.vdocuments.site/reader033/viewer/2022042814/554aba15b4c905ec6c8b46cd/html5/thumbnails/148.jpg)
Common Tools
Gearman
Thursday, 19 May 2011
![Page 149: Scale like an ant, distribute the workload - DPC, Amsterdam, 2011](https://reader033.vdocuments.site/reader033/viewer/2022042814/554aba15b4c905ec6c8b46cd/html5/thumbnails/149.jpg)
Common Tools
Gearman
Hadoop
Thursday, 19 May 2011
![Page 150: Scale like an ant, distribute the workload - DPC, Amsterdam, 2011](https://reader033.vdocuments.site/reader033/viewer/2022042814/554aba15b4c905ec6c8b46cd/html5/thumbnails/150.jpg)
Common Tools
Gearman
Hadoop
Zero MQ (0MQ)
Thursday, 19 May 2011
![Page 151: Scale like an ant, distribute the workload - DPC, Amsterdam, 2011](https://reader033.vdocuments.site/reader033/viewer/2022042814/554aba15b4c905ec6c8b46cd/html5/thumbnails/151.jpg)
Common Tools
Gearman
Hadoop
Zero MQ (0MQ)
RabbitMQ
Thursday, 19 May 2011
![Page 152: Scale like an ant, distribute the workload - DPC, Amsterdam, 2011](https://reader033.vdocuments.site/reader033/viewer/2022042814/554aba15b4c905ec6c8b46cd/html5/thumbnails/152.jpg)
Common Tools
Gearman
Hadoop
Zero MQ (0MQ)
RabbitMQ
And many others!
Thursday, 19 May 2011
![Page 153: Scale like an ant, distribute the workload - DPC, Amsterdam, 2011](https://reader033.vdocuments.site/reader033/viewer/2022042814/554aba15b4c905ec6c8b46cd/html5/thumbnails/153.jpg)
Gearman
Thursday, 19 May 2011
![Page 154: Scale like an ant, distribute the workload - DPC, Amsterdam, 2011](https://reader033.vdocuments.site/reader033/viewer/2022042814/554aba15b4c905ec6c8b46cd/html5/thumbnails/154.jpg)
Your Client Code
Gearman Client API(C, PHP, Perl, MySQL UDF, ...)
Gearman Job Servergearmand
Gearman Worker API(C, PHP, Perl, Python, ...)
Your Worker Code
Your App Gearman
Thursday, 19 May 2011
![Page 155: Scale like an ant, distribute the workload - DPC, Amsterdam, 2011](https://reader033.vdocuments.site/reader033/viewer/2022042814/554aba15b4c905ec6c8b46cd/html5/thumbnails/155.jpg)
A Story!
Thursday, 19 May 2011
![Page 156: Scale like an ant, distribute the workload - DPC, Amsterdam, 2011](https://reader033.vdocuments.site/reader033/viewer/2022042814/554aba15b4c905ec6c8b46cd/html5/thumbnails/156.jpg)
Financial Software
Thursday, 19 May 2011
![Page 157: Scale like an ant, distribute the workload - DPC, Amsterdam, 2011](https://reader033.vdocuments.site/reader033/viewer/2022042814/554aba15b4c905ec6c8b46cd/html5/thumbnails/157.jpg)
3000+ Clients
Thursday, 19 May 2011
![Page 158: Scale like an ant, distribute the workload - DPC, Amsterdam, 2011](https://reader033.vdocuments.site/reader033/viewer/2022042814/554aba15b4c905ec6c8b46cd/html5/thumbnails/158.jpg)
Each one has 5 external data sources
Thursday, 19 May 2011
![Page 159: Scale like an ant, distribute the workload - DPC, Amsterdam, 2011](https://reader033.vdocuments.site/reader033/viewer/2022042814/554aba15b4c905ec6c8b46cd/html5/thumbnails/159.jpg)
Each data source is a web service
Thursday, 19 May 2011
![Page 160: Scale like an ant, distribute the workload - DPC, Amsterdam, 2011](https://reader033.vdocuments.site/reader033/viewer/2022042814/554aba15b4c905ec6c8b46cd/html5/thumbnails/160.jpg)
Ran every 6 hours every day
Thursday, 19 May 2011
![Page 161: Scale like an ant, distribute the workload - DPC, Amsterdam, 2011](https://reader033.vdocuments.site/reader033/viewer/2022042814/554aba15b4c905ec6c8b46cd/html5/thumbnails/161.jpg)
Cron
1
2
3
4
5
Job 1
Gearman
1
43
2
5
Web Services Processing
Thursday, 19 May 2011
![Page 162: Scale like an ant, distribute the workload - DPC, Amsterdam, 2011](https://reader033.vdocuments.site/reader033/viewer/2022042814/554aba15b4c905ec6c8b46cd/html5/thumbnails/162.jpg)
But! That wasn’t enough
Thursday, 19 May 2011
![Page 163: Scale like an ant, distribute the workload - DPC, Amsterdam, 2011](https://reader033.vdocuments.site/reader033/viewer/2022042814/554aba15b4c905ec6c8b46cd/html5/thumbnails/163.jpg)
Job kicked off on login
Thursday, 19 May 2011
![Page 164: Scale like an ant, distribute the workload - DPC, Amsterdam, 2011](https://reader033.vdocuments.site/reader033/viewer/2022042814/554aba15b4c905ec6c8b46cd/html5/thumbnails/164.jpg)
Another Story!
Thursday, 19 May 2011
![Page 165: Scale like an ant, distribute the workload - DPC, Amsterdam, 2011](https://reader033.vdocuments.site/reader033/viewer/2022042814/554aba15b4c905ec6c8b46cd/html5/thumbnails/165.jpg)
CloudSplit
Thursday, 19 May 2011
![Page 166: Scale like an ant, distribute the workload - DPC, Amsterdam, 2011](https://reader033.vdocuments.site/reader033/viewer/2022042814/554aba15b4c905ec6c8b46cd/html5/thumbnails/166.jpg)
Near Real Time Cloud Analytics
Thursday, 19 May 2011
![Page 167: Scale like an ant, distribute the workload - DPC, Amsterdam, 2011](https://reader033.vdocuments.site/reader033/viewer/2022042814/554aba15b4c905ec6c8b46cd/html5/thumbnails/167.jpg)
Clients install logging agent locally
Thursday, 19 May 2011
![Page 168: Scale like an ant, distribute the workload - DPC, Amsterdam, 2011](https://reader033.vdocuments.site/reader033/viewer/2022042814/554aba15b4c905ec6c8b46cd/html5/thumbnails/168.jpg)
syslogd
Thursday, 19 May 2011
![Page 169: Scale like an ant, distribute the workload - DPC, Amsterdam, 2011](https://reader033.vdocuments.site/reader033/viewer/2022042814/554aba15b4c905ec6c8b46cd/html5/thumbnails/169.jpg)
Public API
Thursday, 19 May 2011
![Page 170: Scale like an ant, distribute the workload - DPC, Amsterdam, 2011](https://reader033.vdocuments.site/reader033/viewer/2022042814/554aba15b4c905ec6c8b46cd/html5/thumbnails/170.jpg)
Multiple Persistent Gearman Servers
Thursday, 19 May 2011
![Page 171: Scale like an ant, distribute the workload - DPC, Amsterdam, 2011](https://reader033.vdocuments.site/reader033/viewer/2022042814/554aba15b4c905ec6c8b46cd/html5/thumbnails/171.jpg)
Internal DB API
Thursday, 19 May 2011
![Page 172: Scale like an ant, distribute the workload - DPC, Amsterdam, 2011](https://reader033.vdocuments.site/reader033/viewer/2022042814/554aba15b4c905ec6c8b46cd/html5/thumbnails/172.jpg)
Agent
Thursday, 19 May 2011
![Page 173: Scale like an ant, distribute the workload - DPC, Amsterdam, 2011](https://reader033.vdocuments.site/reader033/viewer/2022042814/554aba15b4c905ec6c8b46cd/html5/thumbnails/173.jpg)
Agent syslogd
Thursday, 19 May 2011
![Page 174: Scale like an ant, distribute the workload - DPC, Amsterdam, 2011](https://reader033.vdocuments.site/reader033/viewer/2022042814/554aba15b4c905ec6c8b46cd/html5/thumbnails/174.jpg)
Agent syslogd
API
Thursday, 19 May 2011
![Page 175: Scale like an ant, distribute the workload - DPC, Amsterdam, 2011](https://reader033.vdocuments.site/reader033/viewer/2022042814/554aba15b4c905ec6c8b46cd/html5/thumbnails/175.jpg)
Agent syslogd
API
Load Balanced
Thursday, 19 May 2011
![Page 176: Scale like an ant, distribute the workload - DPC, Amsterdam, 2011](https://reader033.vdocuments.site/reader033/viewer/2022042814/554aba15b4c905ec6c8b46cd/html5/thumbnails/176.jpg)
Agent syslogd
API
Gearman
Gearman
Load Balanced
Thursday, 19 May 2011
![Page 177: Scale like an ant, distribute the workload - DPC, Amsterdam, 2011](https://reader033.vdocuments.site/reader033/viewer/2022042814/554aba15b4c905ec6c8b46cd/html5/thumbnails/177.jpg)
Agent syslogd
API
Gearman
Gearman
Load Balanced
PersistentThursday, 19 May 2011
![Page 178: Scale like an ant, distribute the workload - DPC, Amsterdam, 2011](https://reader033.vdocuments.site/reader033/viewer/2022042814/554aba15b4c905ec6c8b46cd/html5/thumbnails/178.jpg)
Agent syslogd
API
Gearman
Gearman
Worker
Worker
Worker
Load Balanced
PersistentThursday, 19 May 2011
![Page 179: Scale like an ant, distribute the workload - DPC, Amsterdam, 2011](https://reader033.vdocuments.site/reader033/viewer/2022042814/554aba15b4c905ec6c8b46cd/html5/thumbnails/179.jpg)
Agent syslogd
API
Gearman
Gearman
Worker
Worker
Worker
Internal API
Load Balanced
PersistentThursday, 19 May 2011
![Page 180: Scale like an ant, distribute the workload - DPC, Amsterdam, 2011](https://reader033.vdocuments.site/reader033/viewer/2022042814/554aba15b4c905ec6c8b46cd/html5/thumbnails/180.jpg)
Agent syslogd
API
Gearman
Gearman
Worker
Worker
Worker
Internal API
Load Balanced
Load Balanced
PersistentThursday, 19 May 2011
![Page 181: Scale like an ant, distribute the workload - DPC, Amsterdam, 2011](https://reader033.vdocuments.site/reader033/viewer/2022042814/554aba15b4c905ec6c8b46cd/html5/thumbnails/181.jpg)
Agent syslogd
API
Gearman
Gearman
CouchDB
Worker
Worker
Worker
Internal API
Load Balanced
Load Balanced
PersistentThursday, 19 May 2011
![Page 182: Scale like an ant, distribute the workload - DPC, Amsterdam, 2011](https://reader033.vdocuments.site/reader033/viewer/2022042814/554aba15b4c905ec6c8b46cd/html5/thumbnails/182.jpg)
CouchDB Setup
Thursday, 19 May 2011
![Page 183: Scale like an ant, distribute the workload - DPC, Amsterdam, 2011](https://reader033.vdocuments.site/reader033/viewer/2022042814/554aba15b4c905ec6c8b46cd/html5/thumbnails/183.jpg)
Write vs Read
Thursday, 19 May 2011
![Page 184: Scale like an ant, distribute the workload - DPC, Amsterdam, 2011](https://reader033.vdocuments.site/reader033/viewer/2022042814/554aba15b4c905ec6c8b46cd/html5/thumbnails/184.jpg)
Writes
Thursday, 19 May 2011
![Page 185: Scale like an ant, distribute the workload - DPC, Amsterdam, 2011](https://reader033.vdocuments.site/reader033/viewer/2022042814/554aba15b4c905ec6c8b46cd/html5/thumbnails/185.jpg)
Writes
Multi Master setup
Thursday, 19 May 2011
![Page 186: Scale like an ant, distribute the workload - DPC, Amsterdam, 2011](https://reader033.vdocuments.site/reader033/viewer/2022042814/554aba15b4c905ec6c8b46cd/html5/thumbnails/186.jpg)
Writes
Multi Master setup
Replicated
Thursday, 19 May 2011
![Page 187: Scale like an ant, distribute the workload - DPC, Amsterdam, 2011](https://reader033.vdocuments.site/reader033/viewer/2022042814/554aba15b4c905ec6c8b46cd/html5/thumbnails/187.jpg)
Writes
Multi Master setup
Replicated
Deals with writes only
Thursday, 19 May 2011
![Page 188: Scale like an ant, distribute the workload - DPC, Amsterdam, 2011](https://reader033.vdocuments.site/reader033/viewer/2022042814/554aba15b4c905ec6c8b46cd/html5/thumbnails/188.jpg)
Writes
Multi Master setup
Replicated
Deals with writes only
Thursday, 19 May 2011
![Page 189: Scale like an ant, distribute the workload - DPC, Amsterdam, 2011](https://reader033.vdocuments.site/reader033/viewer/2022042814/554aba15b4c905ec6c8b46cd/html5/thumbnails/189.jpg)
Reads
Thursday, 19 May 2011
![Page 190: Scale like an ant, distribute the workload - DPC, Amsterdam, 2011](https://reader033.vdocuments.site/reader033/viewer/2022042814/554aba15b4c905ec6c8b46cd/html5/thumbnails/190.jpg)
Reads
Multi Master setup
Thursday, 19 May 2011
![Page 191: Scale like an ant, distribute the workload - DPC, Amsterdam, 2011](https://reader033.vdocuments.site/reader033/viewer/2022042814/554aba15b4c905ec6c8b46cd/html5/thumbnails/191.jpg)
Reads
Multi Master setup
Replicated from write cluster
Thursday, 19 May 2011
![Page 192: Scale like an ant, distribute the workload - DPC, Amsterdam, 2011](https://reader033.vdocuments.site/reader033/viewer/2022042814/554aba15b4c905ec6c8b46cd/html5/thumbnails/192.jpg)
Reads
Multi Master setup
Replicated from write cluster
Slaves handle website requests
Thursday, 19 May 2011
![Page 193: Scale like an ant, distribute the workload - DPC, Amsterdam, 2011](https://reader033.vdocuments.site/reader033/viewer/2022042814/554aba15b4c905ec6c8b46cd/html5/thumbnails/193.jpg)
Reads
Multi Master setup
Replicated from write cluster
Slaves handle website requests
Thursday, 19 May 2011
![Page 194: Scale like an ant, distribute the workload - DPC, Amsterdam, 2011](https://reader033.vdocuments.site/reader033/viewer/2022042814/554aba15b4c905ec6c8b46cd/html5/thumbnails/194.jpg)
Heavy Map/Reduce usage for data
Thursday, 19 May 2011
![Page 195: Scale like an ant, distribute the workload - DPC, Amsterdam, 2011](https://reader033.vdocuments.site/reader033/viewer/2022042814/554aba15b4c905ec6c8b46cd/html5/thumbnails/195.jpg)
Supervisord
Thursday, 19 May 2011
![Page 196: Scale like an ant, distribute the workload - DPC, Amsterdam, 2011](https://reader033.vdocuments.site/reader033/viewer/2022042814/554aba15b4c905ec6c8b46cd/html5/thumbnails/196.jpg)
phpadvent.org/2009/daemonize-your-php-by-sean-coates
Thursday, 19 May 2011
![Page 197: Scale like an ant, distribute the workload - DPC, Amsterdam, 2011](https://reader033.vdocuments.site/reader033/viewer/2022042814/554aba15b4c905ec6c8b46cd/html5/thumbnails/197.jpg)
Map/Reduce
Thursday, 19 May 2011
![Page 198: Scale like an ant, distribute the workload - DPC, Amsterdam, 2011](https://reader033.vdocuments.site/reader033/viewer/2022042814/554aba15b4c905ec6c8b46cd/html5/thumbnails/198.jpg)
Map
Thursday, 19 May 2011
![Page 199: Scale like an ant, distribute the workload - DPC, Amsterdam, 2011](https://reader033.vdocuments.site/reader033/viewer/2022042814/554aba15b4c905ec6c8b46cd/html5/thumbnails/199.jpg)
Map
Master gets a problem to solve
Thursday, 19 May 2011
![Page 200: Scale like an ant, distribute the workload - DPC, Amsterdam, 2011](https://reader033.vdocuments.site/reader033/viewer/2022042814/554aba15b4c905ec6c8b46cd/html5/thumbnails/200.jpg)
Map
Master gets a problem to solve
Breaks into multiple sub-problems
Thursday, 19 May 2011
![Page 201: Scale like an ant, distribute the workload - DPC, Amsterdam, 2011](https://reader033.vdocuments.site/reader033/viewer/2022042814/554aba15b4c905ec6c8b46cd/html5/thumbnails/201.jpg)
Map
Master gets a problem to solve
Breaks into multiple sub-problems
Distributed to multiple workers
Thursday, 19 May 2011
![Page 202: Scale like an ant, distribute the workload - DPC, Amsterdam, 2011](https://reader033.vdocuments.site/reader033/viewer/2022042814/554aba15b4c905ec6c8b46cd/html5/thumbnails/202.jpg)
Map
Master gets a problem to solve
Breaks into multiple sub-problems
Distributed to multiple workers
A worker can take the same steps
Thursday, 19 May 2011
![Page 203: Scale like an ant, distribute the workload - DPC, Amsterdam, 2011](https://reader033.vdocuments.site/reader033/viewer/2022042814/554aba15b4c905ec6c8b46cd/html5/thumbnails/203.jpg)
Map
Master gets a problem to solve
Breaks into multiple sub-problems
Distributed to multiple workers
A worker can take the same steps
Answer passed back to Master
Thursday, 19 May 2011
![Page 204: Scale like an ant, distribute the workload - DPC, Amsterdam, 2011](https://reader033.vdocuments.site/reader033/viewer/2022042814/554aba15b4c905ec6c8b46cd/html5/thumbnails/204.jpg)
Reduce
Thursday, 19 May 2011
![Page 205: Scale like an ant, distribute the workload - DPC, Amsterdam, 2011](https://reader033.vdocuments.site/reader033/viewer/2022042814/554aba15b4c905ec6c8b46cd/html5/thumbnails/205.jpg)
Reduce
Takes in answers from the map workers
Thursday, 19 May 2011
![Page 206: Scale like an ant, distribute the workload - DPC, Amsterdam, 2011](https://reader033.vdocuments.site/reader033/viewer/2022042814/554aba15b4c905ec6c8b46cd/html5/thumbnails/206.jpg)
Reduce
Takes in answers from the map workers
Combines together to get an answer
Thursday, 19 May 2011
![Page 207: Scale like an ant, distribute the workload - DPC, Amsterdam, 2011](https://reader033.vdocuments.site/reader033/viewer/2022042814/554aba15b4c905ec6c8b46cd/html5/thumbnails/207.jpg)
Reduce
Takes in answers from the map workers
Combines together to get an answer
There can be multiple reducers
Thursday, 19 May 2011
![Page 208: Scale like an ant, distribute the workload - DPC, Amsterdam, 2011](https://reader033.vdocuments.site/reader033/viewer/2022042814/554aba15b4c905ec6c8b46cd/html5/thumbnails/208.jpg)
process petabytes of data in few hours on commodity server farm
Thursday, 19 May 2011
![Page 209: Scale like an ant, distribute the workload - DPC, Amsterdam, 2011](https://reader033.vdocuments.site/reader033/viewer/2022042814/554aba15b4c905ec6c8b46cd/html5/thumbnails/209.jpg)
CouchDB
Thursday, 19 May 2011
![Page 210: Scale like an ant, distribute the workload - DPC, Amsterdam, 2011](https://reader033.vdocuments.site/reader033/viewer/2022042814/554aba15b4c905ec6c8b46cd/html5/thumbnails/210.jpg)
CouchDB
Thursday, 19 May 2011
![Page 211: Scale like an ant, distribute the workload - DPC, Amsterdam, 2011](https://reader033.vdocuments.site/reader033/viewer/2022042814/554aba15b4c905ec6c8b46cd/html5/thumbnails/211.jpg)
CouchDB
Highly Concurrent
Thursday, 19 May 2011
![Page 212: Scale like an ant, distribute the workload - DPC, Amsterdam, 2011](https://reader033.vdocuments.site/reader033/viewer/2022042814/554aba15b4c905ec6c8b46cd/html5/thumbnails/212.jpg)
CouchDB
Highly Concurrent
Schema free, document based
Thursday, 19 May 2011
![Page 213: Scale like an ant, distribute the workload - DPC, Amsterdam, 2011](https://reader033.vdocuments.site/reader033/viewer/2022042814/554aba15b4c905ec6c8b46cd/html5/thumbnails/213.jpg)
CouchDB
Highly Concurrent
Schema free, document based
RESTful API
Thursday, 19 May 2011
![Page 214: Scale like an ant, distribute the workload - DPC, Amsterdam, 2011](https://reader033.vdocuments.site/reader033/viewer/2022042814/554aba15b4c905ec6c8b46cd/html5/thumbnails/214.jpg)
CouchDB
Highly Concurrent
Schema free, document based
RESTful API
Map/Reduce Views
Thursday, 19 May 2011
![Page 215: Scale like an ant, distribute the workload - DPC, Amsterdam, 2011](https://reader033.vdocuments.site/reader033/viewer/2022042814/554aba15b4c905ec6c8b46cd/html5/thumbnails/215.jpg)
CouchDB
Highly Concurrent
Schema free, document based
RESTful API
Map/Reduce Views
Easy Replication
Thursday, 19 May 2011
![Page 216: Scale like an ant, distribute the workload - DPC, Amsterdam, 2011](https://reader033.vdocuments.site/reader033/viewer/2022042814/554aba15b4c905ec6c8b46cd/html5/thumbnails/216.jpg)
Hadoop
Thursday, 19 May 2011
![Page 217: Scale like an ant, distribute the workload - DPC, Amsterdam, 2011](https://reader033.vdocuments.site/reader033/viewer/2022042814/554aba15b4c905ec6c8b46cd/html5/thumbnails/217.jpg)
Hadoop is a framework for running applications on large clusters of commodity hardware.
Thursday, 19 May 2011
![Page 218: Scale like an ant, distribute the workload - DPC, Amsterdam, 2011](https://reader033.vdocuments.site/reader033/viewer/2022042814/554aba15b4c905ec6c8b46cd/html5/thumbnails/218.jpg)
Thursday, 19 May 2011
![Page 219: Scale like an ant, distribute the workload - DPC, Amsterdam, 2011](https://reader033.vdocuments.site/reader033/viewer/2022042814/554aba15b4c905ec6c8b46cd/html5/thumbnails/219.jpg)
The Hadoop framework transparently provides applications both reliability and data motion
Thursday, 19 May 2011
![Page 220: Scale like an ant, distribute the workload - DPC, Amsterdam, 2011](https://reader033.vdocuments.site/reader033/viewer/2022042814/554aba15b4c905ec6c8b46cd/html5/thumbnails/220.jpg)
Thursday, 19 May 2011
![Page 221: Scale like an ant, distribute the workload - DPC, Amsterdam, 2011](https://reader033.vdocuments.site/reader033/viewer/2022042814/554aba15b4c905ec6c8b46cd/html5/thumbnails/221.jpg)
Uses Map/Reduce concept to farm out work
Thursday, 19 May 2011
![Page 222: Scale like an ant, distribute the workload - DPC, Amsterdam, 2011](https://reader033.vdocuments.site/reader033/viewer/2022042814/554aba15b4c905ec6c8b46cd/html5/thumbnails/222.jpg)
Thursday, 19 May 2011
![Page 223: Scale like an ant, distribute the workload - DPC, Amsterdam, 2011](https://reader033.vdocuments.site/reader033/viewer/2022042814/554aba15b4c905ec6c8b46cd/html5/thumbnails/223.jpg)
Distributed FS to handled node failure automagically
Thursday, 19 May 2011
![Page 224: Scale like an ant, distribute the workload - DPC, Amsterdam, 2011](https://reader033.vdocuments.site/reader033/viewer/2022042814/554aba15b4c905ec6c8b46cd/html5/thumbnails/224.jpg)
Thursday, 19 May 2011
![Page 225: Scale like an ant, distribute the workload - DPC, Amsterdam, 2011](https://reader033.vdocuments.site/reader033/viewer/2022042814/554aba15b4c905ec6c8b46cd/html5/thumbnails/225.jpg)
Join 2 datasets together of a significant size
Thursday, 19 May 2011
![Page 226: Scale like an ant, distribute the workload - DPC, Amsterdam, 2011](https://reader033.vdocuments.site/reader033/viewer/2022042814/554aba15b4c905ec6c8b46cd/html5/thumbnails/226.jpg)
Thursday, 19 May 2011
![Page 227: Scale like an ant, distribute the workload - DPC, Amsterdam, 2011](https://reader033.vdocuments.site/reader033/viewer/2022042814/554aba15b4c905ec6c8b46cd/html5/thumbnails/227.jpg)
500 GB worth of log files with a large location dataset
Thursday, 19 May 2011
![Page 228: Scale like an ant, distribute the workload - DPC, Amsterdam, 2011](https://reader033.vdocuments.site/reader033/viewer/2022042814/554aba15b4c905ec6c8b46cd/html5/thumbnails/228.jpg)
ØMQ
Thursday, 19 May 2011
![Page 229: Scale like an ant, distribute the workload - DPC, Amsterdam, 2011](https://reader033.vdocuments.site/reader033/viewer/2022042814/554aba15b4c905ec6c8b46cd/html5/thumbnails/229.jpg)
ØMQ
Thursday, 19 May 2011
![Page 230: Scale like an ant, distribute the workload - DPC, Amsterdam, 2011](https://reader033.vdocuments.site/reader033/viewer/2022042814/554aba15b4c905ec6c8b46cd/html5/thumbnails/230.jpg)
ØMQ
Async Message System
Thursday, 19 May 2011
![Page 231: Scale like an ant, distribute the workload - DPC, Amsterdam, 2011](https://reader033.vdocuments.site/reader033/viewer/2022042814/554aba15b4c905ec6c8b46cd/html5/thumbnails/231.jpg)
ØMQ
Async Message System
Thin and lightweight
Thursday, 19 May 2011
![Page 232: Scale like an ant, distribute the workload - DPC, Amsterdam, 2011](https://reader033.vdocuments.site/reader033/viewer/2022042814/554aba15b4c905ec6c8b46cd/html5/thumbnails/232.jpg)
ØMQ
Async Message System
Thin and lightweight
High Performance
Thursday, 19 May 2011
![Page 233: Scale like an ant, distribute the workload - DPC, Amsterdam, 2011](https://reader033.vdocuments.site/reader033/viewer/2022042814/554aba15b4c905ec6c8b46cd/html5/thumbnails/233.jpg)
ØMQ
Async Message System
Thin and lightweight
High Performance
Simple
Thursday, 19 May 2011
![Page 234: Scale like an ant, distribute the workload - DPC, Amsterdam, 2011](https://reader033.vdocuments.site/reader033/viewer/2022042814/554aba15b4c905ec6c8b46cd/html5/thumbnails/234.jpg)
ØMQ
Async Message System
Thin and lightweight
High Performance
Simple
Scalable
Thursday, 19 May 2011
![Page 235: Scale like an ant, distribute the workload - DPC, Amsterdam, 2011](https://reader033.vdocuments.site/reader033/viewer/2022042814/554aba15b4c905ec6c8b46cd/html5/thumbnails/235.jpg)
Thursday, 19 May 2011
![Page 236: Scale like an ant, distribute the workload - DPC, Amsterdam, 2011](https://reader033.vdocuments.site/reader033/viewer/2022042814/554aba15b4c905ec6c8b46cd/html5/thumbnails/236.jpg)
One socket can load balance to multiple end points
Thursday, 19 May 2011
![Page 237: Scale like an ant, distribute the workload - DPC, Amsterdam, 2011](https://reader033.vdocuments.site/reader033/viewer/2022042814/554aba15b4c905ec6c8b46cd/html5/thumbnails/237.jpg)
Thursday, 19 May 2011
![Page 238: Scale like an ant, distribute the workload - DPC, Amsterdam, 2011](https://reader033.vdocuments.site/reader033/viewer/2022042814/554aba15b4c905ec6c8b46cd/html5/thumbnails/238.jpg)
Multiple end points can be funnelled into a single socket
Thursday, 19 May 2011
![Page 239: Scale like an ant, distribute the workload - DPC, Amsterdam, 2011](https://reader033.vdocuments.site/reader033/viewer/2022042814/554aba15b4c905ec6c8b46cd/html5/thumbnails/239.jpg)
Thursday, 19 May 2011
![Page 240: Scale like an ant, distribute the workload - DPC, Amsterdam, 2011](https://reader033.vdocuments.site/reader033/viewer/2022042814/554aba15b4c905ec6c8b46cd/html5/thumbnails/240.jpg)
Handle deployments to multiple servers
Thursday, 19 May 2011
![Page 241: Scale like an ant, distribute the workload - DPC, Amsterdam, 2011](https://reader033.vdocuments.site/reader033/viewer/2022042814/554aba15b4c905ec6c8b46cd/html5/thumbnails/241.jpg)
Thursday, 19 May 2011
![Page 242: Scale like an ant, distribute the workload - DPC, Amsterdam, 2011](https://reader033.vdocuments.site/reader033/viewer/2022042814/554aba15b4c905ec6c8b46cd/html5/thumbnails/242.jpg)
Scale is an example of that
Thursday, 19 May 2011
![Page 243: Scale like an ant, distribute the workload - DPC, Amsterdam, 2011](https://reader033.vdocuments.site/reader033/viewer/2022042814/554aba15b4c905ec6c8b46cd/html5/thumbnails/243.jpg)
Thursday, 19 May 2011
![Page 244: Scale like an ant, distribute the workload - DPC, Amsterdam, 2011](https://reader033.vdocuments.site/reader033/viewer/2022042814/554aba15b4c905ec6c8b46cd/html5/thumbnails/244.jpg)
Mongrel2 is a web server that uses it in a similar way as fastcgi
Thursday, 19 May 2011
![Page 245: Scale like an ant, distribute the workload - DPC, Amsterdam, 2011](https://reader033.vdocuments.site/reader033/viewer/2022042814/554aba15b4c905ec6c8b46cd/html5/thumbnails/245.jpg)
Thursday, 19 May 2011
![Page 246: Scale like an ant, distribute the workload - DPC, Amsterdam, 2011](https://reader033.vdocuments.site/reader033/viewer/2022042814/554aba15b4c905ec6c8b46cd/html5/thumbnails/246.jpg)
Move around text (JSON) and Binary data for real time communication
Thursday, 19 May 2011
![Page 247: Scale like an ant, distribute the workload - DPC, Amsterdam, 2011](https://reader033.vdocuments.site/reader033/viewer/2022042814/554aba15b4c905ec6c8b46cd/html5/thumbnails/247.jpg)
Thursday, 19 May 2011
![Page 248: Scale like an ant, distribute the workload - DPC, Amsterdam, 2011](https://reader033.vdocuments.site/reader033/viewer/2022042814/554aba15b4c905ec6c8b46cd/html5/thumbnails/248.jpg)
Could have replaced syslogd and the external API in my previous example
Thursday, 19 May 2011
![Page 249: Scale like an ant, distribute the workload - DPC, Amsterdam, 2011](https://reader033.vdocuments.site/reader033/viewer/2022042814/554aba15b4c905ec6c8b46cd/html5/thumbnails/249.jpg)
Code time? :-)
Thursday, 19 May 2011
![Page 250: Scale like an ant, distribute the workload - DPC, Amsterdam, 2011](https://reader033.vdocuments.site/reader033/viewer/2022042814/554aba15b4c905ec6c8b46cd/html5/thumbnails/250.jpg)
Questions?
Joind.in: http://joind.in/3212
Thursday, 19 May 2011