Download - MPI, Erlang and the web
parallel programmingwhy does it matter to web development
who i am
• german expat in NZ
• text editor addict since age 14
• CTO ideegeo Group Limited
• erlounge Wellington
erlounge Wellington
• diverse crowd
• more about parallel programming
• once a month
todays topics
• message passing infrastructure
• erlang
• open standards
scale web based projects
• one server
• separate functionality (some server)
• load balance (some more server)
loadbalance
• web servers is easy
• databases is harder
• business logic often not possible
the problem
• shared state
• shared data
• waiting for other processes
in general terms
global locking
sounds familiar?looks like a parallel programming issue to me
i tried to solve ...
• scalability issues
• scale to many servers
• asynchronous processes
• process queues
• callback queues
i wrote
• a simple perl based environment
• my own message queue
• my own messaging format
what i actually didwithout knowing it ...
i wrote a MPIok, a quite basic message passing infrastructure but it
was one
and to my big surprise
this thing workedbetter as expected
... but ...
i found erlang
erlang ...
• is a functional language
• has a built in MPI
• share nothing approach
feels like heaven on earth
issues
• not compatible to my existing stuff
• feel the pain of custom protocols
• rewrite or compat layer or ...
the usual thing happened
... i left it aside ...
... meanwhile ...
i started to play with
• CouchDB
• RabbitMQ
• ejabberd
all erlangand open standards based
the more I playedthe more I wished I could use them
the inevitable happened
iteration two
AMQP
open standard
• backed by major players
• around for some years
• RabbitMQ (erlang based)
JSON
open format
• used by many
• small footprint
• easy to read for humans and computers
erlang
fast and small
• compared to perl/php/ruby/...
• scales linear to 32 cores
• distributing processes is easy
perl
still in the system
• for compatibility
• for string processing
• for stuff where it makes sense
[your lang here]
language independent
• JSON is available in most languages
• AMQP is available in all major languages
• pick the one that makes sense
internet architectures
• scaling to many servers
• loadbalancing
• reducing global locks
multicore programming
• scaling to many cores
• loadbalancing
• reducing global locks
do we solve simmilar problems here?
where to find me
• @norbu09
• http://norbu09.org
• http://github.com/norbu09
• http://slideshare.net/norbu09
thanks
• http://www.flickr.com/photos/dranidis/
• http://www.flickr.com/photos/samuellkim/
• http://www.flickr.com/photos/tbcave/