capacity planning for lamp architectures john allspaw manager, operations flickr.com web builder 2.0...

Capacity Planning for LAMP ArchitecturesCapacity Planning for LAMP Architectures

John AllspawJohn AllspawManager, OperationsFlickr.com

Web Builder 2.0 Las Vegas

Capacity Planning for LAMP Capacity Planning for LAMP ArchitecturesArchitectures

Capacity planning:

“the ability to make snap decisions to spend millions of dollars with not enough information”

- Kevin Murphy


It is NOT:

• Performance tuning• Tips and tricks to be “scalable”


It IS:

• What comes after you’ve made it all “scalable”• Making sure that you have enough equipment to

handle gradual and bursty traffic


Questions to answer


How many of each server “class” should you add as you grow ?

Hint: Don’t add too much (too much $$! Ahh!)

Hint: Don’t add too little (too much traffic! Ahh!)


How to make it easier to predict the future* ?

How to make it easier to justify those predictions ?

How to make it easier to predict the future…in the future ?!

*You can’t predict the future, but you can try.


The OLD way of doing things was easier

• A.k.a. “web1.0”

• Small number of content producers

• Control over the content

• Capacity was dictated by the demand for that static/small content

• Even bbs/communities/ecommerce had relatively predictable growth


Today’s way of doing things is harder/fun

• No control over content (users have control)

• No control over usage (users have control)


Today’s way of doing things is harder/fun

• Network effect, nonlinear growth (more users/content/contacts/activity mean >> usage)

• Event-related growth (press, news event, social trend, etc. can affect usage and content)

Example: London bombing, tsunami, holidays, etc.


Considerations for “social” applications

• User behavior should guide you with defining capacity metrics (not just server stats)

• Usage can accelerate, not just grow


How we do it at Flickr


Gathering Usage

Application-level information (users, photos, activity, etc.)

Server-level information (cpu, disk I/O, memory, etc.)

We tie the two together


BEFORE we start collecting server stats

What resources are peak-driven ? (concurrent use)– Ex: photo processed/sec, pages/sec, images/sec, db qps

What resources are permanently consumable ?– Ex: database space, storage (GB/day) etc.


BEFORE we start collecting server stats

What is an average user:consumption ratio ? (example: user: photo)

What is the high and low of ratios ?

Is the average ratio changing over time ?


Non-linear growth


Linear relationships, though


Server and Network statistics

Ganglia - (we love ganglia!)– Multicast-y goodness– SUPER simple to make a graph from any stat– Clustering

Other custom rrdtool-based stuff

MRTG


Photos uploaded/processed/min

Avg processing time per minute

Avg CPU per minute


Gather and record statistics

Accept the ‘observer effect’ (it’s worth it)

Aggregate your stats across clusters

– Stacked graphs– Totals and averages


Squid client requests(24 hours)

(Y axis is req/sec)


Squid LRU reference ageOver 24 hours

-Y axis is “days”-So peak has 3.6hours


Find the ceiling of each class/function/server

Maximum allowable somethings

– MySQL : queries/sec before slave lag sets in

– Apache/php : page requests/sec before total meltdown

– Squid/memcached : cache churn rate, request rate

– Storage : disk I/O utilization, storage limit(!)

– Etc.


Forget benchmarking, use real load

– Make sure you have a easy mechanism to take servers in and out of production

– Pull machines from a balanced pool during medium-level traffic (very carefully)

– Watch and record


Build the infrastructure to make it EASY to measure

Obvious things to help this:

Load balancing Network segmentation Carve up functions into clusters

– Don’t let a machine do more than one primary thing (if you can help it)

this isn’t for performance! If it makes it faster/better, then bonus!


For graphs you don’t have raw data for

GraphClick

http://www.arizona-software.ch/applications/graphclick/en/

- graph digitizer package- $8 US- you pick points on a calibrated image, it spits out tabular data


Once you have:

1. Time history of metrics2. “Ideal” peak levels (ceiling)

Then you can:

3. Predict the future!*


Example:

Photo Processing


Photos uploaded/processed/min

Avg processing time per minute

Avg CPU per minute


Dirty linear math

25% CPU @40 photos/min40% CPU @60 photos/min

So….take a “ceiling”:

75% CPU @112 photos/min = 6720 photos/hour(but double-check the process time)


Conclusion


Know your machines and their limits Measure how the site is being used with application-

level stats Tie real-world observations to server stats


Some Flickr statistics

300M photos, 4 or 5 different sizes Keep ~25M images in cache at any time, ~1M from RAM 2B MySQL queries/day 21k req/sec to memcached 1.2 PT raw disk storage

capacity planning for lamp architectures john allspaw manager, operations flickr.com web builder 2.0...

Documents

lamp architectures server

lamp architectures questions

content capacity

control slide

capacity metrics

minute slide

record slide

scalable slide