Brian M. ShireOpen Web 2008, Vancouver

April 14th 4:45-5:45pm

About Facebook

A social utility to communicate & share information

5th most-trafficked website in the world. (source: comScore)

Over 70 million active users.

Customization.

Global collaboration is beneficial!

“Own the stack.”

Open Source

Facebook uses and modifies Open Source technology.....

Provides a significant PHP interpreter performance increase.

Open source allows collaboration, bug fixes and optimizations.

Alternative PHP Cache (APC)

“APC is a free, open, and robust framework for caching and optimizing PHP intermediate code.”

George SchlossnagleDaniel CowgillRasmus LerdorfGopal VijayaraghavanEdin Kadribasic

APC Developers:

Ilia AlshanetskyMarcus BörgerSara GolemonBrian Shire

Motivations for Optimization

http://www.flickr.com/photos/wwworks/2222523486

Save Energy!

Motivations for Optimization

Save Money!

http://www.flickr.com/photos/twon/1526777999

Motivations for Optimization

Better User Experience!

http://www.flickr.com/photos/dnorman/1539617436

Requests per second provide an overall measurement of performance gains.

Measurements made using Apache Bench (ab) with 1000 total requests, concurrency of 40.Executed on a dual Dual Core AMD Opteron 2.2Ghz, 8GB RAM.

The APC Advantage

Facebook

APC User & File

APC User

PHP 2.4 rps

5.2 rps (x2)

22 rps (x9)

29.5 rps (x12)

Start Request

Compilation:

Convert PHP source

into opcodes

Execute opcodes

End Request

APC hook Opcode

Cache

StoreAPC hook

Miss

Hit

Fetch Opcodes

Start Request

Compilation:

Convert PHP source

into opcodes

Execute opcodes

End Request

PHP

Compilation With Opcode Caches

APC

1 <?php2 $output = ’Hello World’;3 if($_GET[‘exclaim’]) {4 $output .= ‘!’;5 } else {6 $output .= ‘.’;7 }8 echo $output;9 ?>

PHP Source

1 ASSIGN !0, ‘HELLO+WORLD’ 2 FETCH_R GLOBAL $1, ‘_GET’ 3 FETCH_DIM_R $2, $1, ‘exclaim’ 4 JMPZ $2, ->6 5 ASSIGN_CONCAT !0, ‘%21’ 6 JMP ->7 7 ASSIGN_CONCAT !0, ‘.’ 8 ECHO $0 9 RETURN 110 ZEND_HANDLE_EXCEPTION

Opcodes

Opcode output generated with the Vulcan Logic Dissassembler (VLD)by Derick Rethans (http://pecl.php.net/packages/VLD/)

From Code to Opcode

apc.php is a web based console.

Monitor apc.php

Located in apc source directory.

Monitor cache size, hit rate, and configuration.

For automated monitoring, use apc_cache_info() and apc_sma_info() with the $limited flag to reduce load.

Browsing of File and User caches.

Monitor apc.php to determine your memory usage.

Performs poorly when memory is at capacity.

Hints optimize hash lookup tables.

Maximum number of files or user entries.

Cache Size and Hints

apc.shm_segments = 1

Allocate extra memory for overhead.

Deleted items remain in the cache until no longer in use.

Facebook: 1.6GB in production, 6GB in development.

Facebook: over 3,000 files and 490,000 user variables.

apc.shm_size = 30

apc.num_files_hint = 1000

apc.user_entries_hint = 4096

Uses file locking operations

Faster alternative to file locks

Stable, not effecient

Architecture specific, Linux Kernel 2.6.x or later

Significant performance gain

Better alternative to Linux Futex Locks

Same performance gain

Stable, wide OS support

Ported from the PostgreSQL project

Runs in user space

Locking Mechanisms

File Locks

IPC Semaphore Locks

Pthread Mutex Locks

Linux Futex Locks

Spin Locks

Facebook: Spin Locks!

0

50

100

60 120 180

Fil

e

APC Locking Performance Comparison Under High Contention

systemuser

0

50

100

60 120 180

IPC

Sem

. systemuser

0

50

100

60 120 180

Fute

x

systemuser

0

50

100

60 120 180

Pth

read

systemuser

0

50

100

60 120 180

Spin

Time in Seconds

systemuser

APC Locking Performance Comparison Under High Contention

APC stats files to determine if they’ve been updated.Start Request

Compilation:

Convert PHP source

into opcodes

Execute opcodes

End Request

APC hook Opcode

Cache

StoreAPC hook

Miss

Hit

stat()Disabling updates will increase performance.

Requires restart or apc_cache_clear() to update.

CVS, SVN and rsync backdate modified times.

stat_ctime checks the creation time for updates.

To Stat or Not to Stat?

apc.stat = true

apc.stat_ctime = false

Facebook: apc.stat = FALSE!

Inspect System Calls Using strace

System calls are expensive!

Inspect them with strace:

> strace -e trace=file httpd -Xstat("/www/html/test.php", {st_mode=S_IFREG|0644, st_size=710, ...}) = 0getcwd("/usr/local/apache/bin", 4095) = 41chdir("/www/html") = 0open("/www/html/includes/1.inc", O_RDONLY) = 7stat("/www/html/includes/1.inc", {st_mode=S_IFREG|0644, st_size=16, ...}) = 0open("/www/html/includes/2.inc", O_RDONLY) = 8stat("/www/html/includes/2.inc", {st_mode=S_IFREG|0644, st_size=16, ...}) = 0open("/www/html/includes/3.inc", O_RDONLY) = 9stat("/www/html/includes/3.inc", {st_mode=S_IFREG|0644, st_size=16, ...}) = 0open("/www/html/includes/4.inc", O_RDONLY) = 10stat("/www/html/includes/4.inc", {st_mode=S_IFREG|0644, st_size=16, ...}) = 0open("/www/html/includes/5.inc", O_RDONLY) = 11stat("/www/html/includes/5.inc", {st_mode=S_IFREG|0644, st_size=16, ...}) = 0open("/www/html/includes/6.inc", O_RDONLY) = 12stat("/www/html/includes/6.inc", {st_mode=S_IFREG|0644, st_size=16, ...}) = 0open("/www/html/includes/7.inc", O_RDONLY) = 13stat("/www/html/includes/7.inc", {st_mode=S_IFREG|0644, st_size=16, ...}) = 0open("/www/html/includes/8.inc", O_RDONLY) = 14stat("/www/html/includes/8.inc", {st_mode=S_IFREG|0644, st_size=16, ...}) = 0open("/www/html/includes/9.inc", O_RDONLY) = 19stat("/www/html/includes/9.inc", {st_mode=S_IFREG|0644, st_size=16, ...}) = 0open("/www/html/includes/10.inc", O_RDONLY) = 20stat("/www/html/includes/10.inc", {st_mode=S_IFREG|0644, st_size=16, ...}) = 0chdir("/usr/local/apache/bin") = 0

Reducing System CPU

Example:

<?php

/* Include some really important libraries */include_once(‘/www/includes/1.inc’);include_once(‘/www/includes/2.inc’);include_once(‘/www/includes/3.inc’);include_once(‘/www/includes/4.inc’);include_once(‘/www/includes/5.inc’);include_once(‘/www/includes/6.inc’);include_once(‘/www/includes/7.inc’);include_once(‘/www/includes/8.inc’);include_once(‘/www/includes/9.inc’);include_once(‘/www/includes/10.inc’);

echo ‘ok’?>

stat("/www/html/test.php", {st_mode=S_IFREG|0644, st_size=710, ...}) = 0getcwd("/usr/local/apache/bin", 4095) = 41chdir("/www/html") = 0open("/www/html/includes/1.inc", O_RDONLY) = 7open("/www/html/includes/2.inc", O_RDONLY) = 8open("/www/html/includes/3.inc", O_RDONLY) = 9open("/www/html/includes/4.inc", O_RDONLY) = 10open("/www/html/includes/5.inc", O_RDONLY) = 11open("/www/html/includes/6.inc", O_RDONLY) = 12open("/www/html/includes/7.inc", O_RDONLY) = 13open("/www/html/includes/8.inc", O_RDONLY) = 14open("/www/html/includes/9.inc", O_RDONLY) = 19open("/www/html/includes/10.inc", O_RDONLY) = 20chdir("/usr/local/apache/bin") = 0

System Call Reduction With apc.stat=FALSE

stat("/www/html/test.php", {st_mode=S_IFREG|0644, st_size=710, ...}) = 0getcwd("/usr/local/apache/bin", 4095) = 41chdir("/www/html") = 0open("/www/html/includes/1.inc", O_RDONLY) = 7stat("/www/html/includes/1.inc", {st_mode=S_IFREG|0644, st_size=16, ...}) = 0open("/www/html/includes/2.inc", O_RDONLY) = 8stat("/www/html/includes/2.inc", {st_mode=S_IFREG|0644, st_size=16, ...}) = 0open("/www/html/includes/3.inc", O_RDONLY) = 9stat("/www/html/includes/3.inc", {st_mode=S_IFREG|0644, st_size=16, ...}) = 0open("/www/html/includes/4.inc", O_RDONLY) = 10stat("/www/html/includes/4.inc", {st_mode=S_IFREG|0644, st_size=16, ...}) = 0open("/www/html/includes/5.inc", O_RDONLY) = 11stat("/www/html/includes/5.inc", {st_mode=S_IFREG|0644, st_size=16, ...}) = 0open("/www/html/includes/6.inc", O_RDONLY) = 12stat("/www/html/includes/6.inc", {st_mode=S_IFREG|0644, st_size=16, ...}) = 0open("/www/html/includes/7.inc", O_RDONLY) = 13stat("/www/html/includes/7.inc", {st_mode=S_IFREG|0644, st_size=16, ...}) = 0open("/www/html/includes/8.inc", O_RDONLY) = 14stat("/www/html/includes/8.inc", {st_mode=S_IFREG|0644, st_size=16, ...}) = 0open("/www/html/includes/9.inc", O_RDONLY) = 19stat("/www/html/includes/9.inc", {st_mode=S_IFREG|0644, st_size=16, ...}) = 0open("/www/html/includes/10.inc", O_RDONLY) = 20stat("/www/html/includes/10.inc", {st_mode=S_IFREG|0644, st_size=16, ...}) = 0chdir("/usr/local/apache/bin") = 0

apc.stat = TRUE apc.stat = FALSE

Inline garbage collector removes entries from cache as soon as possible.

Spreads load on startup, only caches percent of requests.

The non-blocking write lock is preferred to setting slam_defense.

Read updated files after given delay to prevent loading incomplete files.

“Time to Live”

Entries still in use will be removed after the gc_ttl has expired.

Maximum time a cache entry can remain in cache.

Slam Defenses & TTL

apc.slam_defense = 0

apc.write_lock = 1

apc.file_update_protection = 2

apc.ttl = 0

apc.user_ttl = 0

apc.gc_ttl = 3600

Limits the maximum file size that will be cached.

A regular expression that excludes files from being cached.

Setting to zero causes files to only cache if they match apc.filters.

Filtering Files

apc.max_file_size = 1M

apc.filters = NULL

apc.cache_by_default = true

Monitor Upload Progress

Upload progress support

Update frequency in percent of file size or per byte count

Hidden form input name for apc user variable key

Prefix for the apc user variable key

<form> <input type=”hidden” name=”APC_UPLOAD_PROGRESS” value=”A86DCF0C”> ... </form>

Produces an APC user cache entry “upload_A86DCF0C” containing upload statistics.

The HTML Form:

apc.rfc1867 = On

apc.rfc1867_freq = 0

apc.rfc1867_name = APC_UPLOAD_PROGRESS

apc.rfc1867_prefix = upload_

API

array apc_cache_info ([ string $cache_type [, bool $limited ]] )array apc_sma_info ([ bool $limited ] )

bool apc_store ( string $key , mixed $var [, int $ttl ] )bool apc_add ( string $key , mixed $var [, int $ttl ] )

mixed apc_fetch ( string $key )

bool apc_delete ( string $key )bool apc_clear_cache ([ string $cache_type ] )

bool apc_compile_file ( string $filename )

Info. & Stats.

Insertion

Retrieval

Deletion

File Insertion

User cache stores PHP variables across multiples requests on a per server basis.

Primary commands for utilizing the user cache:

Optimize...Application configuration

Statistics such as site usage, request types, error conditions, timing

Nth tier cache in addition to memcache or other caching service

HTML or other output

Database backed values that only change rarely like product listings

Site behavior like new features, A/B tests, or rate controls

The User Variable Cache

bool apc_store ( string $key , mixed $var [, int $ttl ] )bool apc_add ( string $key , mixed $var [, int $ttl ] )

mixed apc_fetch ( string $key )

Insertion

Retrieval

Control site configuration and features via the user cache.

Datacenter 'B

web Servers:

Datacenter 'A'

Web Servers:

apc_sitevar.php

Spawns requests to multiple web servers

The site updates in the time it takes to make HTTP requests.

Updates are simple and fast for the Engineer.

Engineers Users

Controlling Site Behavior with "Sitevars"

Site-wide Server Variables: “Sitevars”

Facebook primes it’s cache before each server restart.

Ready to serve requests without delay due to compilation or updating cache values.

Handles immediate flood of requests with limited warm-up time.

Starts with same cache state limiting differences between servers.

Improve Startup with Cache Priming

Apache is restarted for major code pushes.

bool apc_store ( string $key , mixed $var [, int $ttl ] )

bool apc_compile_file ( string $filename )

Insertion

File Insertion

Serialize Cache Values

Stop Apache

Disallow connections via

iptables or load balancer

Deserialize and apc_store()

values.

Start Apache

apc_compile_file()

PHP source

Allow connections via

iptables or load balancer

Serialized User

Cache Values

Serialized User

Cache Values

Distributed to Web Servers

Master Restart Script

Single Control Server Multiple Web Servers

System

Apache

PHP & APC

Local Disk

APC Priming and Restart

APC Binary Dump (experimental patch)

Allows the abilty to copy the APC memory cache to disk, and reload at a later time.

Ability to load pre-compiled files into your cache.

Ability to load and run code without touching disk or keeping source code on file.

Does not currently support apc.stat=1 mode.

Only works between identical architectures (x86_64 vs. i386, little vs. big endian).

Works only between the same PHP versions.

Prime APC Cache and

apc_bindump_file("gzip://...")

Stop Apache

Disallow connections via

iptables or load balancer

apc_binload_file("gzip://...")

Start Apache

Allow connections via

iptables or load balancer

Binary File Binary FileDistributed to Web Servers

Master Restart Script

Single Control Server Multiple Web Servers

System

Apache

PHP & APC

Local Disk

APC Binary Priming and Restart

Lazy Loading

Significant time is spent copying opcodes from APC.

Only a small portion of the code is executed.

Delay copying until function is called.

Perceived as include() being slow.

Will soon be available as an experimental patch to APC and PHP.

include('library.php');

library_function();

exit();

<?php // Normal LoadingCopyALL

Functions

APCFunctions

ExecuteSingle

Function

include('library.php');

library_function();

exit();

<?php // Lazy Loading

APCFunctions

Copy Single

Function&

Execute

Installation via “pecl install apc”or source http://pecl.php.net/packages/APC/

Start with a basic apc.ini file:

apc.enabled=1apc.shm_size=100M

Install apc.php under the DocumentRoot, configure the USER and PASS variables.Monitor apc.php for usage and adjust configuration.

Try tuning some of the discussed settings to meet application needs.

Add some APC user variables.

Try spin locks.

Try the apc.stat=0 setting.

Measure changes in CPU usage and maximum requests per second.

Getting Started

Further Information...

I appreciate feedback/questions about APC, patches, this talk, and anything else! <shire@facebook.com>

More implementation examples in Lucas Nealan's “Facebook Performance Caching”http://sizzo.org/wp/talks/

Handout, slides, and patches will be available at http://tekrat.com

Acknowledgments

George SchlossnagleDaniel CowgillRasmus LerdorfGopal VijayaraghavanEdin Kadribasic

APC Developers:

Ilia AlshanetskyMarcus BörgerSara Golemon

Thanks to the Open Web Conference organizers!

Any Questions?

Q & A

Thank you!