evented applications with rabbitmq and cakephp
DESCRIPTION
Talk given at CakeFest 2011 about using RabbitMQ, CakePHP and pyhon to create soft realtime applications.TRANSCRIPT
![Page 1: Evented applications with RabbitMQ and CakePHP](https://reader033.vdocuments.site/reader033/viewer/2022061604/5552c160b4c905920f8b48bd/html5/thumbnails/1.jpg)
Evented Applications
with RabbitMQ , Python, and CakePHP
![Page 2: Evented applications with RabbitMQ and CakePHP](https://reader033.vdocuments.site/reader033/viewer/2022061604/5552c160b4c905920f8b48bd/html5/thumbnails/2.jpg)
What the wibblefish?
All about offloading work, and doing things in ‘realtime’
Make applications that complete tasks in a distributed/scalable fashion.
Doing work outside the request/response cycle of a UI/API.
![Page 3: Evented applications with RabbitMQ and CakePHP](https://reader033.vdocuments.site/reader033/viewer/2022061604/5552c160b4c905920f8b48bd/html5/thumbnails/3.jpg)
Work to offload
Noti"cations (SMS, Email, Postal mail).
Logging.
Import & export.
Process image & video.
Reporting.
Webhooks.
![Page 4: Evented applications with RabbitMQ and CakePHP](https://reader033.vdocuments.site/reader033/viewer/2022061604/5552c160b4c905920f8b48bd/html5/thumbnails/4.jpg)
Isn’t cron good enough?
Cron tabs are great with one server.
Cron doesn’t scale to multiple machines.
Cron doesn’t solve the whole problem space.
![Page 5: Evented applications with RabbitMQ and CakePHP](https://reader033.vdocuments.site/reader033/viewer/2022061604/5552c160b4c905920f8b48bd/html5/thumbnails/5.jpg)
Problems with Cron
Isn’t suitable for ‘real-time’ results.
Requires a set schedule. You might spin up processes that do nothing. Or you might not have enough capacity.
Parallelizing is a pain.
![Page 6: Evented applications with RabbitMQ and CakePHP](https://reader033.vdocuments.site/reader033/viewer/2022061604/5552c160b4c905920f8b48bd/html5/thumbnails/6.jpg)
Queues to the rescue!
Queues accept and dispatch events.
Events can contain almost anything*.
Events can be roughly equal to ‘jobs’.
A single event can trigger lots of downstream work.
![Page 7: Evented applications with RabbitMQ and CakePHP](https://reader033.vdocuments.site/reader033/viewer/2022061604/5552c160b4c905920f8b48bd/html5/thumbnails/7.jpg)
Architectural benefits
Separation of concerns.
Collection of mini/small applications.
No monolithic applications.
Decoupled, and easier to test.
Easy to scale. Busy service = more of them.
![Page 8: Evented applications with RabbitMQ and CakePHP](https://reader033.vdocuments.site/reader033/viewer/2022061604/5552c160b4c905920f8b48bd/html5/thumbnails/8.jpg)
Queue technologies
RabbitMQ
ZeroMQ
SQS
Celery
Gearman
![Page 9: Evented applications with RabbitMQ and CakePHP](https://reader033.vdocuments.site/reader033/viewer/2022061604/5552c160b4c905920f8b48bd/html5/thumbnails/9.jpg)
RabbitMQ
Written in Erlang.
Open source and Free.
Really Fast.
Socket protocol that is has cheap connections.
Data persists after a crash.
![Page 10: Evented applications with RabbitMQ and CakePHP](https://reader033.vdocuments.site/reader033/viewer/2022061604/5552c160b4c905920f8b48bd/html5/thumbnails/10.jpg)
Queue basics
Producers create events.
Consumers eat messages, and do work.
messages == events.
![Page 11: Evented applications with RabbitMQ and CakePHP](https://reader033.vdocuments.site/reader033/viewer/2022061604/5552c160b4c905920f8b48bd/html5/thumbnails/11.jpg)
Queue Basics
Producer
![Page 12: Evented applications with RabbitMQ and CakePHP](https://reader033.vdocuments.site/reader033/viewer/2022061604/5552c160b4c905920f8b48bd/html5/thumbnails/12.jpg)
Queue Basics
Producer
![Page 13: Evented applications with RabbitMQ and CakePHP](https://reader033.vdocuments.site/reader033/viewer/2022061604/5552c160b4c905920f8b48bd/html5/thumbnails/13.jpg)
Queue Basics
Producer Consumer
![Page 14: Evented applications with RabbitMQ and CakePHP](https://reader033.vdocuments.site/reader033/viewer/2022061604/5552c160b4c905920f8b48bd/html5/thumbnails/14.jpg)
Queue Basics
Producer
Consumer
Consumer
![Page 15: Evented applications with RabbitMQ and CakePHP](https://reader033.vdocuments.site/reader033/viewer/2022061604/5552c160b4c905920f8b48bd/html5/thumbnails/15.jpg)
Queue Basics
Producer
Consumer
Consumer
Consumer
![Page 16: Evented applications with RabbitMQ and CakePHP](https://reader033.vdocuments.site/reader033/viewer/2022061604/5552c160b4c905920f8b48bd/html5/thumbnails/16.jpg)
Exchanges
Exchanges abstract queues from the producer.
Route messages to queues, that are bound by routing keys.
Allows consumers to listen to speci"c routing keys.
Multiple consumers can listen to the same events.
![Page 17: Evented applications with RabbitMQ and CakePHP](https://reader033.vdocuments.site/reader033/viewer/2022061604/5552c160b4c905920f8b48bd/html5/thumbnails/17.jpg)
Exchanges
Producer
![Page 18: Evented applications with RabbitMQ and CakePHP](https://reader033.vdocuments.site/reader033/viewer/2022061604/5552c160b4c905920f8b48bd/html5/thumbnails/18.jpg)
Exchanges
Producer
![Page 19: Evented applications with RabbitMQ and CakePHP](https://reader033.vdocuments.site/reader033/viewer/2022061604/5552c160b4c905920f8b48bd/html5/thumbnails/19.jpg)
Exchanges
Producer
Consumer
page_view
![Page 20: Evented applications with RabbitMQ and CakePHP](https://reader033.vdocuments.site/reader033/viewer/2022061604/5552c160b4c905920f8b48bd/html5/thumbnails/20.jpg)
Exchanges
ProducerConsumerimport
Consumer
page_view
![Page 21: Evented applications with RabbitMQ and CakePHP](https://reader033.vdocuments.site/reader033/viewer/2022061604/5552c160b4c905920f8b48bd/html5/thumbnails/21.jpg)
Exchanges
Producer
Consumererror,
page_view
Consumerimport
Consumer
page_view
![Page 22: Evented applications with RabbitMQ and CakePHP](https://reader033.vdocuments.site/reader033/viewer/2022061604/5552c160b4c905920f8b48bd/html5/thumbnails/22.jpg)
Topic subscription
Consumers can also listen to topics. Which are keys separated by dots.
e.g page_view.search, page_view.update
Consumer could listen to page_view.*
![Page 23: Evented applications with RabbitMQ and CakePHP](https://reader033.vdocuments.site/reader033/viewer/2022061604/5552c160b4c905920f8b48bd/html5/thumbnails/23.jpg)
Producing
Can create messages with many tools.
PHP, python, node, ruby, etc.
Messages can contain any stringy data (JSON)
![Page 24: Evented applications with RabbitMQ and CakePHP](https://reader033.vdocuments.site/reader033/viewer/2022061604/5552c160b4c905920f8b48bd/html5/thumbnails/24.jpg)
Data for messages
Only string-y things can be stored in messages.
JSON encoded data in message.
Key to value stored in Redis/Memcache.
Keep messages small. RabbitMQ performs better with small messages.
![Page 25: Evented applications with RabbitMQ and CakePHP](https://reader033.vdocuments.site/reader033/viewer/2022061604/5552c160b4c905920f8b48bd/html5/thumbnails/25.jpg)
Consuming
Use a daemon process.
Daemons in PHP can be done, but they kinda suck.
Usually use sleep(2) or other workaround.
sleep(2) moves you away from ‘real time’.
![Page 26: Evented applications with RabbitMQ and CakePHP](https://reader033.vdocuments.site/reader033/viewer/2022061604/5552c160b4c905920f8b48bd/html5/thumbnails/26.jpg)
Aw bugger..
![Page 27: Evented applications with RabbitMQ and CakePHP](https://reader033.vdocuments.site/reader033/viewer/2022061604/5552c160b4c905920f8b48bd/html5/thumbnails/27.jpg)
Use a better tool
Other tools are much better for making daemons.
Python (pika, kombu, sparkplug)
NodeJS (node-amqp)
Ruby (carrot, ruby-amqp)
![Page 28: Evented applications with RabbitMQ and CakePHP](https://reader033.vdocuments.site/reader033/viewer/2022061604/5552c160b4c905920f8b48bd/html5/thumbnails/28.jpg)
Managing mixed applications
Many ways to get it done.
Avoid duplicating code.
Thin daemons + worker processes.
![Page 29: Evented applications with RabbitMQ and CakePHP](https://reader033.vdocuments.site/reader033/viewer/2022061604/5552c160b4c905920f8b48bd/html5/thumbnails/29.jpg)
Thin daemons
Python/node daemon accepts messages.
Delegate to PHP worker processes.
Keeps PHP’s fork + die work#ow.
Re-use application code by calling shells.
![Page 30: Evented applications with RabbitMQ and CakePHP](https://reader033.vdocuments.site/reader033/viewer/2022061604/5552c160b4c905920f8b48bd/html5/thumbnails/30.jpg)
Thin daemons
PHP
![Page 31: Evented applications with RabbitMQ and CakePHP](https://reader033.vdocuments.site/reader033/viewer/2022061604/5552c160b4c905920f8b48bd/html5/thumbnails/31.jpg)
Thin daemons
PHP
![Page 32: Evented applications with RabbitMQ and CakePHP](https://reader033.vdocuments.site/reader033/viewer/2022061604/5552c160b4c905920f8b48bd/html5/thumbnails/32.jpg)
Thin daemons
PHP
Py
![Page 33: Evented applications with RabbitMQ and CakePHP](https://reader033.vdocuments.site/reader033/viewer/2022061604/5552c160b4c905920f8b48bd/html5/thumbnails/33.jpg)
Thin daemons
PHP Py
Py
![Page 34: Evented applications with RabbitMQ and CakePHP](https://reader033.vdocuments.site/reader033/viewer/2022061604/5552c160b4c905920f8b48bd/html5/thumbnails/34.jpg)
Thin daemons
PHP Py
Py
Py
![Page 35: Evented applications with RabbitMQ and CakePHP](https://reader033.vdocuments.site/reader033/viewer/2022061604/5552c160b4c905920f8b48bd/html5/thumbnails/35.jpg)
Thin daemons
PHP Py
Py
Py PHP
![Page 36: Evented applications with RabbitMQ and CakePHP](https://reader033.vdocuments.site/reader033/viewer/2022061604/5552c160b4c905920f8b48bd/html5/thumbnails/36.jpg)
Thin daemons
PHP Py PHP
Py
Py PHP
![Page 37: Evented applications with RabbitMQ and CakePHP](https://reader033.vdocuments.site/reader033/viewer/2022061604/5552c160b4c905920f8b48bd/html5/thumbnails/37.jpg)
Thin daemons
PHP Py PHP
Py PHP
Py PHP
![Page 38: Evented applications with RabbitMQ and CakePHP](https://reader033.vdocuments.site/reader033/viewer/2022061604/5552c160b4c905920f8b48bd/html5/thumbnails/38.jpg)
Demo Time
![Page 39: Evented applications with RabbitMQ and CakePHP](https://reader033.vdocuments.site/reader033/viewer/2022061604/5552c160b4c905920f8b48bd/html5/thumbnails/39.jpg)
Overview
Use CakeLog to "re events.
Use CakePHP shell’s for actual work.
Thin python daemon for processing messages.
![Page 40: Evented applications with RabbitMQ and CakePHP](https://reader033.vdocuments.site/reader033/viewer/2022061604/5552c160b4c905920f8b48bd/html5/thumbnails/40.jpg)
Python + sparkplug
Python module for creating queue listeners.
Created by a developer at FreshBooks.
Ini con"guration "le de"nes behaviour.
![Page 41: Evented applications with RabbitMQ and CakePHP](https://reader033.vdocuments.site/reader033/viewer/2022061604/5552c160b4c905920f8b48bd/html5/thumbnails/41.jpg)
What we’ll try:
User events logged during application.
Log events to the database.
![Page 42: Evented applications with RabbitMQ and CakePHP](https://reader033.vdocuments.site/reader033/viewer/2022061604/5552c160b4c905920f8b48bd/html5/thumbnails/42.jpg)
Insert code
![Page 43: Evented applications with RabbitMQ and CakePHP](https://reader033.vdocuments.site/reader033/viewer/2022061604/5552c160b4c905920f8b48bd/html5/thumbnails/43.jpg)
Limitations of using shells
Daemon and shell processes have to be on the same box.
This can be limiting as you scale, depends on how you structure applications.
![Page 44: Evented applications with RabbitMQ and CakePHP](https://reader033.vdocuments.site/reader033/viewer/2022061604/5552c160b4c905920f8b48bd/html5/thumbnails/44.jpg)
Solutions
Use HTTP.
Have thin daemons dispatch back to the front-end application using HTTP.
Use Basic/Digest authentication.
Use checksums.
![Page 45: Evented applications with RabbitMQ and CakePHP](https://reader033.vdocuments.site/reader033/viewer/2022061604/5552c160b4c905920f8b48bd/html5/thumbnails/45.jpg)
HTTP daemons
PHP
![Page 46: Evented applications with RabbitMQ and CakePHP](https://reader033.vdocuments.site/reader033/viewer/2022061604/5552c160b4c905920f8b48bd/html5/thumbnails/46.jpg)
HTTP daemons
PHP
![Page 47: Evented applications with RabbitMQ and CakePHP](https://reader033.vdocuments.site/reader033/viewer/2022061604/5552c160b4c905920f8b48bd/html5/thumbnails/47.jpg)
HTTP daemons
PHP Py
![Page 48: Evented applications with RabbitMQ and CakePHP](https://reader033.vdocuments.site/reader033/viewer/2022061604/5552c160b4c905920f8b48bd/html5/thumbnails/48.jpg)
HTTP daemons
PHP Py
Web
![Page 49: Evented applications with RabbitMQ and CakePHP](https://reader033.vdocuments.site/reader033/viewer/2022061604/5552c160b4c905920f8b48bd/html5/thumbnails/49.jpg)
HTTP daemons
PHP Py Web
Web
![Page 50: Evented applications with RabbitMQ and CakePHP](https://reader033.vdocuments.site/reader033/viewer/2022061604/5552c160b4c905920f8b48bd/html5/thumbnails/50.jpg)
HTTP daemons
PHP Py Web
Web
Web
![Page 51: Evented applications with RabbitMQ and CakePHP](https://reader033.vdocuments.site/reader033/viewer/2022061604/5552c160b4c905920f8b48bd/html5/thumbnails/51.jpg)
Thanks; Questions?