Celery for internal API in SOA infrastructure

Roman ImankulovPyCon Russia, Feb 2013

1. Internal API. What's the deal

SOA service architecture example

1. Internal API. What's the deal

● DCOM: too Microsoftish

● CORBA: too Enterprisy

● SOAP: XML inside

● RESTful: too sloppy

Celery to the rescue?

Agenda

1. Internal API. What's the deal

2. How to use Celery for internal API

3. Organizing Celery-based API. Routing

4. Celery-based API. Benefits, pitfalls and

security notes

2. How to use Celery for internal API

Celery Worker

2. How to use Celery for internal API

Classic celery client imports task function

2. How to use Celery for internal API

Classic celery client imports makes you:● install everything

○ slows down installation

● import everything○ python processes grow in size○ circular dependencies○ tight coupling

● share everything○ ... with outsource developers

2. How to use Celery for internal API

"Task implementation is not to be exposed without necessity"

William of Ockham

2. How to use Celery for internal API

Low-level interface defines only task name

2. How to use Celery for internal API

Convenient API requires some boilerplate code

2. How to use Celery for internal API

Make it even more convenient with introspection

2. How to use Celery for internal API

Introspection magic with celery-api

http://github.com/imankulov/celery-api

3. Celery-based API. Routing

3. Celery-based API. Routing

Creating API endpoint

1. Common package for all API endpoints2. Every worker imports celery instance3. Expose tasks for worker with

CELERY_IMPORTS

4. Launch worker withcelery worker -A module_name -Q foo

-n foo

3. Celery-based API. Routing

Send tasks with

add.apply_async(queue='foo', args=(1, 1))

3. Celery-based API. Routing

Define routing rules

3. Celery-based API. Routing

Task in the wire

{ "body": <base64 encoded string with task name and args>, "properties": { "body_encoding": "base64", "delivery_info": { "priority": 0, "routing_key": "foo", "exchange": "foo" }, "delivery_mode": 2, "delivery_tag": <UUID> }, ...}

3. Celery-based API. Routing

Task in the wire

"delivery_info": { "priority": 0, "routing_key": "foo", "exchange": "foo" }

3. Celery-based API. Routing

AMQP for dummies

3. Celery-based API. Routing

Define all queues (API endpoints) you have

4. Celery-based API. Benefits

High level protocol.

Takes the responsibility for

data serialization routingexception handling

4. Celery-based API. Benefits

Support for wide range of brokers.

You can switch between brokers with no major changes in your code

Brokers:RabbitMQ Redis SQLAlchemy Django MongoDB Amazon SQSCouchDB Beanstalk

4. Celery-based API. Benefits

Built-in extra stuff which you were afraid to ever dream of.

asynchronous execution parallel executionasynchronous parallel executiondelayed execution throttlingautomatic retrying of failed tasks limiting the time of executionautoscaling API inspectionpublic key cryptography (message signing)

4. Celery-based API. Pitfalls

Python-centric.

Celery is written in Python and meant to be used with Python code exclusively

4. Celery-based API. Pitfalls

Increasing complexity.

Celery codebase 46k LOCKombu codebase 19k LOC

CompareFlask codebase 10k LOCWerkzeug codebase 28k LOCJinja2 codebase 16k LOC

4. Celery-based API. Security note

Never ever expose Celery broker to the Web!

4. Celery-based API. Security note

1. Try >>> redis_url = 'redis://:pycon2013@broker.imankulov.name'>>> celery = Celery(broker=redis_url, result=redis_url)

2. Get the contents of /etc/pycon_secret

3. Exchange it for a secret prize ;)