mpi, erlang and the web
DESCRIPTION
slides from my talk at LCA2010 in Wellington about how to scale web services and why parallel programming and open standards matterTRANSCRIPT
parallel programmingwhy does it matter to web development
who i am
• german expat in NZ
• text editor addict since age 14
• CTO ideegeo Group Limited
• erlounge Wellington
erlounge Wellington
• diverse crowd
• more about parallel programming
• once a month
todays topics
• message passing infrastructure
• erlang
• open standards
scale web based projects
• one server
• separate functionality (some server)
• load balance (some more server)
loadbalance
• web servers is easy
• databases is harder
• business logic often not possible
the problem
• shared state
• shared data
• waiting for other processes
in general terms
global locking
sounds familiar?looks like a parallel programming issue to me
i tried to solve ...
• scalability issues
• scale to many servers
• asynchronous processes
• process queues
• callback queues
i wrote
• a simple perl based environment
• my own message queue
• my own messaging format
what i actually didwithout knowing it ...
i wrote a MPIok, a quite basic message passing infrastructure but it
was one
and to my big surprise
this thing workedbetter as expected
... but ...
i found erlang
erlang ...
• is a functional language
• has a built in MPI
• share nothing approach
feels like heaven on earth
issues
• not compatible to my existing stuff
• feel the pain of custom protocols
• rewrite or compat layer or ...
the usual thing happened
... i left it aside ...
... meanwhile ...
i started to play with
• CouchDB
• RabbitMQ
• ejabberd
all erlangand open standards based
the more I playedthe more I wished I could use them
the inevitable happened
iteration two
AMQP
open standard
• backed by major players
• around for some years
• RabbitMQ (erlang based)
JSON
open format
• used by many
• small footprint
• easy to read for humans and computers
erlang
fast and small
• compared to perl/php/ruby/...
• scales linear to 32 cores
• distributing processes is easy
perl
still in the system
• for compatibility
• for string processing
• for stuff where it makes sense
[your lang here]
language independent
• JSON is available in most languages
• AMQP is available in all major languages
• pick the one that makes sense
internet architectures
• scaling to many servers
• loadbalancing
• reducing global locks
multicore programming
• scaling to many cores
• loadbalancing
• reducing global locks
do we solve simmilar problems here?
where to find me
• @norbu09
• http://norbu09.org
• http://github.com/norbu09
• http://slideshare.net/norbu09
thanks
• http://www.flickr.com/photos/dranidis/
• http://www.flickr.com/photos/samuellkim/
• http://www.flickr.com/photos/tbcave/