concurrency and distributed systems using jruby
DESCRIPTION
Presented at JRubyConf EU August 2012TRANSCRIPT
CONCURRENCYAND DISTRIBUTED SYSTEMS IN JRUBY
@iconara
speakerdeck.com/u/iconara(real time!)
Theo / @iconara
Chief Architect at
big data with Ruby since 2009
MAKING THIS
INTO THIS
big data with Ruby since 2009
APRIL 2011stuck
MAY 2011let’s try JRuby
JUNE 2011wrapping Java libraries for fun and for profit
HOTBUNNIESnatural Ruby API for RabbitMQ
JULY 2011threads are better than processes
JRUBY
JRUBY
J.U.C(java.util.concurrent)
SHORTCUTJava Concurrency in Practice
EXECUTORSconfigurable thread pools, a better
abstraction than using Thread directly
thread_pool = Executors.new_fixed_thread_pool(16)
all_the_things.each do |thing| thread_pool.submit do crunch_crunch_crunch(thing) endend
BLOCKING QUEUESeasy producer/consumer patterns
J.U.C.ATOMICoptimistic locking primitives,
compare-and-swap, thread safe non-blocking counters
MORE AWESOMESConcurrentHashMap, CountDownLatch,
ForkJoinPool, Google Guava, LMAX distruptor
TL; DRuse j.u.c and avoid mutable state
and you’re safe, mostly
enough ropeTHREADS
AUGUST 2011threads are a mess, let’s try actors
ACTORSa higher level abstraction of concurrency
AKKAconcurrency library for Scala,famous actor implementation
MIKKAAkka wrapper for JRuby
DO ALL THE THINGSProgramming Concurrency on the JVM
AUGUST 2011async systems with no back pressure
mechanism will die of memory starvation
STATE
AGGREGATION
INPUT
PERSISTENCE
OUTPUT
TIMER
STATE
AGGREGATION
INPUT
PERSISTENCE
OUTPUT
TIMER
!
AUGUST 2011actors are cool, but not really
worth it when all the arrows in your flow chart point in the same direction
DIVIDE DIVIDE DIVIDErun multiple independent, synchronous workers,
tune to the number of CPUs & IO wait
SEPTEMBER 2011blocking all the way down,
back pressure all the way up
AUGUST 2011(short rant about MongoDB’s global write lock)
NEW SHINY TOYSfor building new shiny products
CASSANDRAdistributed database, mostly magic
CASSANDRAvery verbose drivers
ColumnFamilyUpdater<String, String> updater = template.createUpdater("a key");updater.setString("domain", "www.datastax.com");updater.setLong("time", System.currentTimeMillis());
try { template.update(updater);} catch (HectorException e) { // hurgh}
EURYDICEJRuby nails it
new_data = {:domain => 'www.datastax.com', :time => Time.now.to_i}column_family.update('a key', new_data)
STORMstream processing framework written in Clojure
STORM(it does the arrows, you do the blobs)
REDSTORMStorm for JRuby
DON’T DO THIS AT HOME KIDS!the key to building concurrent and distributed
systems is to separate what you want to do from the concurrent- and distributed-ness, and let someone
else take care of that
YOUR CODE
JRUBY IS A SLIPPERY SLOPE TO JBOSS
let there be no doubt about it
SCALING IS HARDgo to Srdan’s talk tomorrow
KTHXBAI@iconara
github.com/iconaraarchitecturalatrocities.com
burtcorp.com