all the little pieces
TRANSCRIPT
![Page 1: All The Little Pieces](https://reader034.vdocuments.site/reader034/viewer/2022052619/5553563eb4c905cf188b466a/html5/thumbnails/1.jpg)
all the little piecesdistributed systems with PHP
Andrei Zmievski ⁜ DiggOSCON 2009 ⁜ San Jose
twitter: @a
Thursday, July 23, 2009
![Page 2: All The Little Pieces](https://reader034.vdocuments.site/reader034/viewer/2022052619/5553563eb4c905cf188b466a/html5/thumbnails/2.jpg)
Who is this guy?
• Open Source Fellow @ Digg
• PHP Core Developer since 1999
• Architect of the Unicode/i18n support
• Release Manager for PHP 6
• Beer lover (and brewer)
Thursday, July 23, 2009
![Page 3: All The Little Pieces](https://reader034.vdocuments.site/reader034/viewer/2022052619/5553563eb4c905cf188b466a/html5/thumbnails/3.jpg)
Why distributed?
• Because Moore’s Law will not save you
• Despite what DHH says
Thursday, July 23, 2009
![Page 4: All The Little Pieces](https://reader034.vdocuments.site/reader034/viewer/2022052619/5553563eb4c905cf188b466a/html5/thumbnails/4.jpg)
Share nothing
• Your Mom was wrong
• No shared data on application servers
• Distribute it to shared systems
Thursday, July 23, 2009
![Page 5: All The Little Pieces](https://reader034.vdocuments.site/reader034/viewer/2022052619/5553563eb4c905cf188b466a/html5/thumbnails/5.jpg)
distribute…
• memory (memcached)
• storage (mogilefs)
• work (gearman)
Thursday, July 23, 2009
![Page 6: All The Little Pieces](https://reader034.vdocuments.site/reader034/viewer/2022052619/5553563eb4c905cf188b466a/html5/thumbnails/6.jpg)
Building blocks
• GLAMMP - have you heard of it?
• Gearman + LAMP + Memcached
• Throw in Mogile too
Thursday, July 23, 2009
![Page 7: All The Little Pieces](https://reader034.vdocuments.site/reader034/viewer/2022052619/5553563eb4c905cf188b466a/html5/thumbnails/7.jpg)
memcachedThursday, July 23, 2009
![Page 8: All The Little Pieces](https://reader034.vdocuments.site/reader034/viewer/2022052619/5553563eb4c905cf188b466a/html5/thumbnails/8.jpg)
background
• created by Danga Interactive
• high-performance, distributed memory object caching system
• sustains Digg, Facebook, LiveJournal, Yahoo!, and many others
• if you aren’t using it, you are crazy
Thursday, July 23, 2009
![Page 9: All The Little Pieces](https://reader034.vdocuments.site/reader034/viewer/2022052619/5553563eb4c905cf188b466a/html5/thumbnails/9.jpg)
background
• Very fast over the network and very easy to set up
• Designed to be transient
• You still need a database
Thursday, July 23, 2009
![Page 10: All The Little Pieces](https://reader034.vdocuments.site/reader034/viewer/2022052619/5553563eb4c905cf188b466a/html5/thumbnails/10.jpg)
architecture
client
client
client
memcached
memcached
memcached
Thursday, July 23, 2009
![Page 11: All The Little Pieces](https://reader034.vdocuments.site/reader034/viewer/2022052619/5553563eb4c905cf188b466a/html5/thumbnails/11.jpg)
architecture
memcached
Thursday, July 23, 2009
![Page 12: All The Little Pieces](https://reader034.vdocuments.site/reader034/viewer/2022052619/5553563eb4c905cf188b466a/html5/thumbnails/12.jpg)
slab #1
slab #2
slab #3
slab #4
152 bytes
152 bytes
456 bytes
1368 bytes
152 bytes
4104 bytes
152 bytes
152 bytes
456 bytes456 bytes
1368 bytes
Thursday, July 23, 2009
![Page 13: All The Little Pieces](https://reader034.vdocuments.site/reader034/viewer/2022052619/5553563eb4c905cf188b466a/html5/thumbnails/13.jpg)
memory architecture
• memory allocated on startup, released on shutdown
• variable sized slabs (30+ by default)
• each object is stored in the slab most fitting its size
• fragmentation can be problematic
Thursday, July 23, 2009
![Page 14: All The Little Pieces](https://reader034.vdocuments.site/reader034/viewer/2022052619/5553563eb4c905cf188b466a/html5/thumbnails/14.jpg)
memory architecture
• items are deleted:
• on set
• on get, if it’s expired
• if slab is full, then use LRU
Thursday, July 23, 2009
![Page 15: All The Little Pieces](https://reader034.vdocuments.site/reader034/viewer/2022052619/5553563eb4c905cf188b466a/html5/thumbnails/15.jpg)
applications
• object cache
• output cache
• action flood control / rate limiting
• simple queue
• and much more
Thursday, July 23, 2009
![Page 16: All The Little Pieces](https://reader034.vdocuments.site/reader034/viewer/2022052619/5553563eb4c905cf188b466a/html5/thumbnails/16.jpg)
PHP clients
• a few private ones (Facebook, Yahoo!, etc)
• pecl/memcache
• pecl/memcached
Thursday, July 23, 2009
![Page 17: All The Little Pieces](https://reader034.vdocuments.site/reader034/viewer/2022052619/5553563eb4c905cf188b466a/html5/thumbnails/17.jpg)
pecl/memcached
• based on libmemcached
• released in January 2009
• surface API similarity to pecl/memcache
• parity with other languages
Thursday, July 23, 2009
![Page 18: All The Little Pieces](https://reader034.vdocuments.site/reader034/viewer/2022052619/5553563eb4c905cf188b466a/html5/thumbnails/18.jpg)
• get
• set
• add
• replace
• delete
• append
• prepend
• cas
• *_by_key
• getMulti
• setMulti
• getDelayed / fetch*
• callbacks
API
Thursday, July 23, 2009
![Page 19: All The Little Pieces](https://reader034.vdocuments.site/reader034/viewer/2022052619/5553563eb4c905cf188b466a/html5/thumbnails/19.jpg)
consistent hashing
A
B
IP2-1
IP1
IP3
IP2-2
Thursday, July 23, 2009
![Page 20: All The Little Pieces](https://reader034.vdocuments.site/reader034/viewer/2022052619/5553563eb4c905cf188b466a/html5/thumbnails/20.jpg)
compare-and-swap (cas)
• “check and set”
• no update if object changed
• relies on CAS token
Thursday, July 23, 2009
![Page 21: All The Little Pieces](https://reader034.vdocuments.site/reader034/viewer/2022052619/5553563eb4c905cf188b466a/html5/thumbnails/21.jpg)
compare-and-swap (cas)$m = new Memcached();$m->addServer('localhost', 11211);
do { $ips = $m->get('ip_block', null, $cas);
if ($m->getResultCode() == Memcached::RES_NOTFOUND) {
$ips = array($_SERVER['REMOTE_ADDR']); $m->add('ip_block', $ips);
} else {
$ips[] = $_SERVER['REMOTE_ADDR']; $m->cas($cas, 'ip_block', $ips); }
} while ($m->getResultCode() != Memcached::RES_SUCCESS);
Thursday, July 23, 2009
![Page 22: All The Little Pieces](https://reader034.vdocuments.site/reader034/viewer/2022052619/5553563eb4c905cf188b466a/html5/thumbnails/22.jpg)
delayed “lazy” fetching
• issue request with getDelayed()
• do other work
• fetch results with fetch() or fetchAll()
Thursday, July 23, 2009
![Page 23: All The Little Pieces](https://reader034.vdocuments.site/reader034/viewer/2022052619/5553563eb4c905cf188b466a/html5/thumbnails/23.jpg)
binary protocol
• performance• every request is parsed• can happen thousands times a
second
• extensibility
• support more data in the protocol
Thursday, July 23, 2009
![Page 24: All The Little Pieces](https://reader034.vdocuments.site/reader034/viewer/2022052619/5553563eb4c905cf188b466a/html5/thumbnails/24.jpg)
callbacks
• read-through cache callback
• if key is not found, invoke callback, save value to memcache and return it
Thursday, July 23, 2009
![Page 25: All The Little Pieces](https://reader034.vdocuments.site/reader034/viewer/2022052619/5553563eb4c905cf188b466a/html5/thumbnails/25.jpg)
callbacks
• result callback
• invoked by getDelayed() for every found item
• should not call fetch() in this case
Thursday, July 23, 2009
![Page 26: All The Little Pieces](https://reader034.vdocuments.site/reader034/viewer/2022052619/5553563eb4c905cf188b466a/html5/thumbnails/26.jpg)
buffered writes
• queue up write requests
• send when a threshold is exceeded or a ‘get’ command is issued
Thursday, July 23, 2009
![Page 27: All The Little Pieces](https://reader034.vdocuments.site/reader034/viewer/2022052619/5553563eb4c905cf188b466a/html5/thumbnails/27.jpg)
key prefixing
• optional prefix prepended to all the keys automatically
• allows for namespacing, versioning, etc.
Thursday, July 23, 2009
![Page 28: All The Little Pieces](https://reader034.vdocuments.site/reader034/viewer/2022052619/5553563eb4c905cf188b466a/html5/thumbnails/28.jpg)
key locality
• allows mapping a set of keys to a specific server
Thursday, July 23, 2009
![Page 29: All The Little Pieces](https://reader034.vdocuments.site/reader034/viewer/2022052619/5553563eb4c905cf188b466a/html5/thumbnails/29.jpg)
multiple serializers
• PHP
• igbinary
• JSON (soon)
Thursday, July 23, 2009
![Page 30: All The Little Pieces](https://reader034.vdocuments.site/reader034/viewer/2022052619/5553563eb4c905cf188b466a/html5/thumbnails/30.jpg)
future
• UDP support
• replication
• server management (ejection, status callback)
Thursday, July 23, 2009
![Page 31: All The Little Pieces](https://reader034.vdocuments.site/reader034/viewer/2022052619/5553563eb4c905cf188b466a/html5/thumbnails/31.jpg)
tips & tricks
• 32-bit systems with > 4GB memory:
memcached -m4096 -p11211memcached -m4096 -p11212memcached -m4096 -p11213
Thursday, July 23, 2009
![Page 32: All The Little Pieces](https://reader034.vdocuments.site/reader034/viewer/2022052619/5553563eb4c905cf188b466a/html5/thumbnails/32.jpg)
tips & tricks
• write-through or write-back cache
• Warm up the cache on code push
• Version the keys (if necessary)
Thursday, July 23, 2009
![Page 33: All The Little Pieces](https://reader034.vdocuments.site/reader034/viewer/2022052619/5553563eb4c905cf188b466a/html5/thumbnails/33.jpg)
tips & tricks
• Don’t think row-level DB-style caching; think complex objects
• Don’t run memcached on your DB server — your DBAs might send you threatening notes
• Use multi-get — run things in parallel
Thursday, July 23, 2009
![Page 34: All The Little Pieces](https://reader034.vdocuments.site/reader034/viewer/2022052619/5553563eb4c905cf188b466a/html5/thumbnails/34.jpg)
delete by namespace
1 $ns_key = $memcache->get("foo_namespace_key");2 // if not set, initialize it3 if ($ns_key === false)4 $memcache->set("foo_namespace_key",5 rand(1, 10000));6 // cleverly use the ns_key7 $my_key = "foo_".$ns_key."_12345";8 $my_val = $memcache->get($my_key);9 // to clear the namespace:10 $memcache->increment("foo_namespace_key");
Thursday, July 23, 2009
![Page 35: All The Little Pieces](https://reader034.vdocuments.site/reader034/viewer/2022052619/5553563eb4c905cf188b466a/html5/thumbnails/35.jpg)
storing lists of data
• Store items under indexed keys: comment.12, comment.23, etc
• Then store the list of item IDs in another key: comments
• To retrieve, fetch comments and then multi-get the comment IDs
Thursday, July 23, 2009
![Page 36: All The Little Pieces](https://reader034.vdocuments.site/reader034/viewer/2022052619/5553563eb4c905cf188b466a/html5/thumbnails/36.jpg)
preventing stampeding
• embedded probabilistic timeout
• gearman unique task trick
Thursday, July 23, 2009
![Page 37: All The Little Pieces](https://reader034.vdocuments.site/reader034/viewer/2022052619/5553563eb4c905cf188b466a/html5/thumbnails/37.jpg)
optimization
• watch stats (eviction rate, fill, etc)
• getStats()
• telnet + “stats” commands
• peep (heap inspector)
Thursday, July 23, 2009
![Page 38: All The Little Pieces](https://reader034.vdocuments.site/reader034/viewer/2022052619/5553563eb4c905cf188b466a/html5/thumbnails/38.jpg)
slabs
• Tune slab sizes to your needs:
• -f chunk size growth factor (default 1.25)
• -n minimum space allocated for key+value+flags (default 48)
Thursday, July 23, 2009
![Page 39: All The Little Pieces](https://reader034.vdocuments.site/reader034/viewer/2022052619/5553563eb4c905cf188b466a/html5/thumbnails/39.jpg)
slabs
slab class 1: chunk size 104 perslab 10082slab class 2: chunk size 136 perslab 7710slab class 3: chunk size 176 perslab 5957slab class 4: chunk size 224 perslab 4681...slab class 38: chunk size 394840 perslab 2slab class 39: chunk size 493552 perslab 2
Default: 38 slabs
Thursday, July 23, 2009
![Page 40: All The Little Pieces](https://reader034.vdocuments.site/reader034/viewer/2022052619/5553563eb4c905cf188b466a/html5/thumbnails/40.jpg)
slabs
slab class 1: chunk size 1048 perslab 1000slab class 2: chunk size 1064 perslab 985slab class 3: chunk size 1080 perslab 970slab class 4: chunk size 1096 perslab 956...slab class 198: chunk size 9224 perslab 113slab class 199: chunk size 9320 perslab 112
Most objects: ∼1-2KB, some larger
memcached -n 1000 -f 1.01
Thursday, July 23, 2009
![Page 41: All The Little Pieces](https://reader034.vdocuments.site/reader034/viewer/2022052619/5553563eb4c905cf188b466a/html5/thumbnails/41.jpg)
memcached @ digg
Thursday, July 23, 2009
![Page 42: All The Little Pieces](https://reader034.vdocuments.site/reader034/viewer/2022052619/5553563eb4c905cf188b466a/html5/thumbnails/42.jpg)
ops
• memcached on each app server (2GB)
• the process is niced to a lower level
• separate pool for sessions
• 2 servers keep track of cluster health
Thursday, July 23, 2009
![Page 43: All The Little Pieces](https://reader034.vdocuments.site/reader034/viewer/2022052619/5553563eb4c905cf188b466a/html5/thumbnails/43.jpg)
key prefixes
• global key prefix for apc, memcached, etc
• each pool has additional, versioned prefix: .sess.2
• the key version is incremented on each release
• global prefix can invalidate all caches
Thursday, July 23, 2009
![Page 44: All The Little Pieces](https://reader034.vdocuments.site/reader034/viewer/2022052619/5553563eb4c905cf188b466a/html5/thumbnails/44.jpg)
cache chain
• multi-level caching: globals, APC, memcached, etc.
• all cache access is through Cache_Chain class
• various configurations:• APC ➡ memcached• $GLOBALS ➡ APC
Thursday, July 23, 2009
![Page 45: All The Little Pieces](https://reader034.vdocuments.site/reader034/viewer/2022052619/5553563eb4c905cf188b466a/html5/thumbnails/45.jpg)
other
• large objects (> 1MB)
• split on the client side
• save the partial keys in a master one
Thursday, July 23, 2009
![Page 46: All The Little Pieces](https://reader034.vdocuments.site/reader034/viewer/2022052619/5553563eb4c905cf188b466a/html5/thumbnails/46.jpg)
stats
Thursday, July 23, 2009
![Page 47: All The Little Pieces](https://reader034.vdocuments.site/reader034/viewer/2022052619/5553563eb4c905cf188b466a/html5/thumbnails/47.jpg)
moxi - memcached proxy
• homogenization
• multi-get escalation
• protocol pipelining
• statistics, etc
Thursday, July 23, 2009
![Page 48: All The Little Pieces](https://reader034.vdocuments.site/reader034/viewer/2022052619/5553563eb4c905cf188b466a/html5/thumbnails/48.jpg)
alternatives
• in-memory: Tokyo Tyrant, Scalaris
• persistent: Hypertable, Cassandra, MemcacheDB
• document-oriented: CouchDB
Thursday, July 23, 2009
![Page 49: All The Little Pieces](https://reader034.vdocuments.site/reader034/viewer/2022052619/5553563eb4c905cf188b466a/html5/thumbnails/49.jpg)
mogileThursday, July 23, 2009
![Page 50: All The Little Pieces](https://reader034.vdocuments.site/reader034/viewer/2022052619/5553563eb4c905cf188b466a/html5/thumbnails/50.jpg)
background
• created by Danga Interactive
• application-level distributed filesystem
• used at Digg, LiveJournal, etc
• a form of “cloud caching”
• scales very well
Thursday, July 23, 2009
![Page 51: All The Little Pieces](https://reader034.vdocuments.site/reader034/viewer/2022052619/5553563eb4c905cf188b466a/html5/thumbnails/51.jpg)
background
• automatic file replication with custom policies
• no single point of failure
• flat namespace
• local filesystem agnostic
• not meant for speed
Thursday, July 23, 2009
![Page 52: All The Little Pieces](https://reader034.vdocuments.site/reader034/viewer/2022052619/5553563eb4c905cf188b466a/html5/thumbnails/52.jpg)
architecture
tracker
nodeapp tracker DB
node
node
node
tracker
node
Thursday, July 23, 2009
![Page 53: All The Little Pieces](https://reader034.vdocuments.site/reader034/viewer/2022052619/5553563eb4c905cf188b466a/html5/thumbnails/53.jpg)
applications
• images
• document storage
• backing store for certain caches
Thursday, July 23, 2009
![Page 54: All The Little Pieces](https://reader034.vdocuments.site/reader034/viewer/2022052619/5553563eb4c905cf188b466a/html5/thumbnails/54.jpg)
PHP client
• File_Mogile in PEAR
• MediaWiki one (not maintained)
Thursday, July 23, 2009
![Page 55: All The Little Pieces](https://reader034.vdocuments.site/reader034/viewer/2022052619/5553563eb4c905cf188b466a/html5/thumbnails/55.jpg)
Example
$hosts = array('172.10.1.1', '172.10.1.2');$m = new File_Mogile($hosts, 'profiles');$m->storeFile('user1234', 'image', '/tmp/image1234.jpg');
...
$paths = $m->getPaths('user1234');
Thursday, July 23, 2009
![Page 56: All The Little Pieces](https://reader034.vdocuments.site/reader034/viewer/2022052619/5553563eb4c905cf188b466a/html5/thumbnails/56.jpg)
mogile @ digg
Thursday, July 23, 2009
![Page 57: All The Little Pieces](https://reader034.vdocuments.site/reader034/viewer/2022052619/5553563eb4c905cf188b466a/html5/thumbnails/57.jpg)
mogile @ digg
• Wrapper around File_Mogile to cache entries in memcache
• fairly standard set-up
• trackers run on storage nodes
Thursday, July 23, 2009
![Page 58: All The Little Pieces](https://reader034.vdocuments.site/reader034/viewer/2022052619/5553563eb4c905cf188b466a/html5/thumbnails/58.jpg)
mogile @ digg
• not huge (about 3.5 TB of data)
• files are replicated 3x
• the user profile images are cached on Netscaler (1.5 GB cache)
• mogile cluster load is light
Thursday, July 23, 2009
![Page 59: All The Little Pieces](https://reader034.vdocuments.site/reader034/viewer/2022052619/5553563eb4c905cf188b466a/html5/thumbnails/59.jpg)
gearmanThursday, July 23, 2009
![Page 60: All The Little Pieces](https://reader034.vdocuments.site/reader034/viewer/2022052619/5553563eb4c905cf188b466a/html5/thumbnails/60.jpg)
background
• created by Danga Interactive
• anagram of “manager”
• a system for distributing work
• a form of RPC mechanism
Thursday, July 23, 2009
![Page 61: All The Little Pieces](https://reader034.vdocuments.site/reader034/viewer/2022052619/5553563eb4c905cf188b466a/html5/thumbnails/61.jpg)
background
• parallel, asynchronous, scales well
• fire and forget, decentralized
• avoid tying up Apache processes
Thursday, July 23, 2009
![Page 62: All The Little Pieces](https://reader034.vdocuments.site/reader034/viewer/2022052619/5553563eb4c905cf188b466a/html5/thumbnails/62.jpg)
background
• dispatch function calls to machines that are better suited to do work
• do work in parallel
• load balance lots of function calls
• invoke functions in other languages
Thursday, July 23, 2009
![Page 63: All The Little Pieces](https://reader034.vdocuments.site/reader034/viewer/2022052619/5553563eb4c905cf188b466a/html5/thumbnails/63.jpg)
architecture
gearmandclient
workerclient
client
worker
worker
Thursday, July 23, 2009
![Page 64: All The Little Pieces](https://reader034.vdocuments.site/reader034/viewer/2022052619/5553563eb4c905cf188b466a/html5/thumbnails/64.jpg)
applications
• thumbnail generation
• asynchronous logging
• cache warm-up
• DB jobs, data migration
• sending email
Thursday, July 23, 2009
![Page 65: All The Little Pieces](https://reader034.vdocuments.site/reader034/viewer/2022052619/5553563eb4c905cf188b466a/html5/thumbnails/65.jpg)
servers
• Gearman-Server (Perl)
• gearmand (C)
Thursday, July 23, 2009
![Page 66: All The Little Pieces](https://reader034.vdocuments.site/reader034/viewer/2022052619/5553563eb4c905cf188b466a/html5/thumbnails/66.jpg)
clients
• Net_Gearman
• simplified, pretty stable
• pecl/gearman
• more powerful, complex, somewhat unstable (under development)
Thursday, July 23, 2009
![Page 67: All The Little Pieces](https://reader034.vdocuments.site/reader034/viewer/2022052619/5553563eb4c905cf188b466a/html5/thumbnails/67.jpg)
Concepts
• Job
• Worker
• Task
• Client
Thursday, July 23, 2009
![Page 68: All The Little Pieces](https://reader034.vdocuments.site/reader034/viewer/2022052619/5553563eb4c905cf188b466a/html5/thumbnails/68.jpg)
Net_Gearman
• Net_Gearman_Job
• Net_Gearman_Worker
• Net_Gearman_Task
• Net_Gearman_Set
• Net_Gearman_Client
Thursday, July 23, 2009
![Page 69: All The Little Pieces](https://reader034.vdocuments.site/reader034/viewer/2022052619/5553563eb4c905cf188b466a/html5/thumbnails/69.jpg)
Echo Job
class Net_Gearman_Job_Echo extends Net_Gearman_Job_Common{ public function run($arg) { var_export($arg); echo "\n"; }} Echo.php
Thursday, July 23, 2009
![Page 70: All The Little Pieces](https://reader034.vdocuments.site/reader034/viewer/2022052619/5553563eb4c905cf188b466a/html5/thumbnails/70.jpg)
Reverse Jobclass Net_Gearman_Job_Reverse extends Net_Gearman_Job_Common{ public function run($arg) { $result = array(); $n = count($arg); $i = 0; while ($value = array_pop($arg)) { $result[] = $value; $i++; $this->status($i, $n); }
return $result; }} Reverse.php
Thursday, July 23, 2009
![Page 71: All The Little Pieces](https://reader034.vdocuments.site/reader034/viewer/2022052619/5553563eb4c905cf188b466a/html5/thumbnails/71.jpg)
Worker
define('NET_GEARMAN_JOB_PATH', './');
require 'Net/Gearman/Worker.php';
try { $worker = new Net_Gearman_Worker(array('localhost:4730')); $worker->addAbility('Reverse'); $worker->addAbility('Echo'); $worker->beginWork();} catch (Net_Gearman_Exception $e) { echo $e->getMessage() . "\n"; exit;}
Thursday, July 23, 2009
![Page 72: All The Little Pieces](https://reader034.vdocuments.site/reader034/viewer/2022052619/5553563eb4c905cf188b466a/html5/thumbnails/72.jpg)
Client
require_once 'Net/Gearman/Client.php';
function complete($job, $handle, $result) { echo "$job complete, result: ".var_export($result, true)."\n";}
function status($job, $handle, $n, $d){ echo "$n/$d\n";} continued..
Thursday, July 23, 2009
![Page 73: All The Little Pieces](https://reader034.vdocuments.site/reader034/viewer/2022052619/5553563eb4c905cf188b466a/html5/thumbnails/73.jpg)
Client
$client = new Net_Gearman_Client(array('lager:4730'));
$task = new Net_Gearman_Task('Reverse', range(1,5));$task->attachCallback("complete",Net_Gearman_Task::TASK_COMPLETE);$task->attachCallback("status",Net_Gearman_Task::TASK_STATUS);
continued..
Thursday, July 23, 2009
![Page 74: All The Little Pieces](https://reader034.vdocuments.site/reader034/viewer/2022052619/5553563eb4c905cf188b466a/html5/thumbnails/74.jpg)
Client
$set = new Net_Gearman_Set();$set->addTask($task);
$client->runSet($set);
$client->Echo('Mmm... beer');
Thursday, July 23, 2009
![Page 75: All The Little Pieces](https://reader034.vdocuments.site/reader034/viewer/2022052619/5553563eb4c905cf188b466a/html5/thumbnails/75.jpg)
pecl/gearman
• More complex API
• Jobs aren’t separated into files
Thursday, July 23, 2009
![Page 76: All The Little Pieces](https://reader034.vdocuments.site/reader034/viewer/2022052619/5553563eb4c905cf188b466a/html5/thumbnails/76.jpg)
Worker$gmworker= new gearman_worker();$gmworker->add_server();$gmworker->add_function("reverse", "reverse_fn");
while (1){ $ret= $gmworker->work(); if ($ret != GEARMAN_SUCCESS) break;}
function reverse_fn($job){ $workload= $job->workload(); echo "Received job: " . $job->handle() . "\n"; echo "Workload: $workload\n"; $result= strrev($workload); echo "Result: $result\n"; return $result;}
Thursday, July 23, 2009
![Page 77: All The Little Pieces](https://reader034.vdocuments.site/reader034/viewer/2022052619/5553563eb4c905cf188b466a/html5/thumbnails/77.jpg)
Client
$gmclient= new gearman_client();
$gmclient->add_server('lager');
echo "Sending job\n";
list($ret, $result) = $gmclient->do("reverse", "Hello!");
if ($ret == GEARMAN_SUCCESS) echo "Success: $result\n";
Thursday, July 23, 2009
![Page 78: All The Little Pieces](https://reader034.vdocuments.site/reader034/viewer/2022052619/5553563eb4c905cf188b466a/html5/thumbnails/78.jpg)
gearman @ digg
Thursday, July 23, 2009
![Page 79: All The Little Pieces](https://reader034.vdocuments.site/reader034/viewer/2022052619/5553563eb4c905cf188b466a/html5/thumbnails/79.jpg)
gearman @ digg
• 400,000 jobs a day
• Jobs: crawling, DB job, FB sync, memcache manipulation, Twitter post, IDDB migration, etc.
• Each application server has its own Gearman daemon + workers
Thursday, July 23, 2009
![Page 80: All The Little Pieces](https://reader034.vdocuments.site/reader034/viewer/2022052619/5553563eb4c905cf188b466a/html5/thumbnails/80.jpg)
tips and tricks
• you can daemonize the workers easily with daemon or supervisord
• run workers in different groups, don’t block on job A waiting on job B
• Make workers exit after N jobs to free up memory (supervisord will restart them)
Thursday, July 23, 2009
![Page 81: All The Little Pieces](https://reader034.vdocuments.site/reader034/viewer/2022052619/5553563eb4c905cf188b466a/html5/thumbnails/81.jpg)
ThriftThursday, July 23, 2009
![Page 82: All The Little Pieces](https://reader034.vdocuments.site/reader034/viewer/2022052619/5553563eb4c905cf188b466a/html5/thumbnails/82.jpg)
background
• NOT developed by Danga (Facebook)
• cross-language services
• RPC-based
Thursday, July 23, 2009
![Page 83: All The Little Pieces](https://reader034.vdocuments.site/reader034/viewer/2022052619/5553563eb4c905cf188b466a/html5/thumbnails/83.jpg)
background
• interface description language
• bindings: C++, C#, Cocoa, Erlang, Haskell, Java, OCaml, Perl, PHP, Python, Ruby, Smalltalk
• data types: base, structs, constants, services, exceptions
Thursday, July 23, 2009
![Page 84: All The Little Pieces](https://reader034.vdocuments.site/reader034/viewer/2022052619/5553563eb4c905cf188b466a/html5/thumbnails/84.jpg)
IDL
Thursday, July 23, 2009
![Page 85: All The Little Pieces](https://reader034.vdocuments.site/reader034/viewer/2022052619/5553563eb4c905cf188b466a/html5/thumbnails/85.jpg)
Demo
Thursday, July 23, 2009
![Page 86: All The Little Pieces](https://reader034.vdocuments.site/reader034/viewer/2022052619/5553563eb4c905cf188b466a/html5/thumbnails/86.jpg)
Thank You
http://gravitonic.com/talksThursday, July 23, 2009