making symfony services async with rabbitmq (and more symfony)
TRANSCRIPT
It’s all about eXperience
Gaetano Giunta
PHP LondonJuly 2015
Making Symfony Services async with
RabbitMq
(and more Symfony)
PechaKucha style!
2
A case study: generating M$Office docs
The needs
• Produce content in Microsoft Office formats• Many formats for each document
• Both Word and Excel docs
• XML content generated by CMS
• LibreOffice used to generate MS Office and PDF versions• Has anyone tried generating OOXML by hand?
3
A case study: generating M$Office docs
Reality bites
• Slooow• Rest assured that some of the docs will be pretty big
• Unreliable• Depending on version, LO crashes from a-bit to almost-
always
• Race-prone• Two conversion jobs in parallel might step on each other
4
A case study: generating M$Office docs
…and the results are:
• Ugly code: lots of polling, copying of files around, manual locking
• Does not scale at all: only one conversion process active at any given time
Web server
PHP LibreOffice
Waiting processes
11
The rationale
• Same library handling both sides of the connection• Easier to troubleshoot
• Sf Console component is lovely
• Everything developed in a single repository
• No need to learn new languages
• Everything which I can rewrite in PHP, I eventually will*
* = 3rd whimsical law of Gaetano
12
The details
• Json used as payload encoding• The worker has been made thread-safe
• Again, thanks to Symfony: the « Process » Component• It spins up a new php process each time it receives a command• The new process executes the command
• Bonus: a ‘watchdog’ to restart workers via crontab if they die
Symfonylistener
Symfonylistener
Symfonylistener
Symfonyworker
LibreOffice
LibreOffice
W
13
The evolution
• What is exactly a « queued command » ?
• For starters, each existing Symfony console command!
• Why not every existing Symfony Service?• Allows to run existing services asynchronously
• As long as their arguments are serializable to json
• Not a project-specific tool any more
• Could be turned into a Bundle
• …
• Profit!
14
The problems
• Assumes good security on the network
• Low throughput• Spinning up a new php worker process takes time
• Could we use php-fpm to communicate to workers instead?
• Am I reinventing the wheel?
• Others?• Suggestions are welcome