cooking pies with celery
TRANSCRIPT
![Page 1: Cooking pies with Celery](https://reader031.vdocuments.site/reader031/viewer/2022031922/5882bcf81a28abb2478b5279/html5/thumbnails/1.jpg)
Печём пирожки с Celery
Александр Мокров
![Page 2: Cooking pies with Celery](https://reader031.vdocuments.site/reader031/viewer/2022031922/5882bcf81a28abb2478b5279/html5/thumbnails/2.jpg)
О чем доклад
Немного об очередях
Краткий обзор возможностей Celery
Использование на примере виртуальной фабрики пирожков
![Page 3: Cooking pies with Celery](https://reader031.vdocuments.site/reader031/viewer/2022031922/5882bcf81a28abb2478b5279/html5/thumbnails/3.jpg)
Очереди
Producer
queue_name
Consumer
![Page 4: Cooking pies with Celery](https://reader031.vdocuments.site/reader031/viewer/2022031922/5882bcf81a28abb2478b5279/html5/thumbnails/4.jpg)
Очереди
Producer
Consumer
Consumer
queue_name
![Page 5: Cooking pies with Celery](https://reader031.vdocuments.site/reader031/viewer/2022031922/5882bcf81a28abb2478b5279/html5/thumbnails/5.jpg)
Роутинг
Producer
Consumer
Consumer
first_queue
Xsecond_queue
white
white
black
![Page 6: Cooking pies with Celery](https://reader031.vdocuments.site/reader031/viewer/2022031922/5882bcf81a28abb2478b5279/html5/thumbnails/6.jpg)
What is task queue?
![Page 7: Cooking pies with Celery](https://reader031.vdocuments.site/reader031/viewer/2022031922/5882bcf81a28abb2478b5279/html5/thumbnails/7.jpg)
Celery: Distributed Task QueueCelery is an asynchronous task queue/job queue based on distributed message passing. It is focused on real-time operation, but supports scheduling as well.
The execution units, called tasks, are executed concurrently on a single or more worker servers using multiprocessing, Eventlet, or gevent. Tasks can execute asynchronously (in the background) or synchronously (wait until ready).
Celery is used in production systems to process millions of tasks a day.
![Page 8: Cooking pies with Celery](https://reader031.vdocuments.site/reader031/viewer/2022031922/5882bcf81a28abb2478b5279/html5/thumbnails/8.jpg)
![Page 9: Cooking pies with Celery](https://reader031.vdocuments.site/reader031/viewer/2022031922/5882bcf81a28abb2478b5279/html5/thumbnails/9.jpg)
Ask Solem Hoelhttps://github.com/ask
Principal Software Engineer at Robinhood
San Francisco, CA
Staff Engineer, RabbitMQVMwareмай 2011 – май 2013 (2 года 1 месяц)
Open source and consulting work on Celery, RabbitMQ,
Celery Дата начала: май 2009
![Page 10: Cooking pies with Celery](https://reader031.vdocuments.site/reader031/viewer/2022031922/5882bcf81a28abb2478b5279/html5/thumbnails/10.jpg)
CeleryKombu (Messaging library for Python) 41991 (22541) lines
Billiard (Python multiprocessing fork with improvements and bugfixes) 19191(13115)
Vine (promise, async, future) 2921
Celery 104296 (37495)
Total 168399(76072)
![Page 11: Cooking pies with Celery](https://reader031.vdocuments.site/reader031/viewer/2022031922/5882bcf81a28abb2478b5279/html5/thumbnails/11.jpg)
It supportsBrokers
RabbitMQ, Redis,
MongoDB (exp), ZeroMQ (exp)
CouchDB (exp), SQLAlchemy (exp)
Django ORM (exp), Amazon SQS, (exp)
Concurrency
prefork (multiprocessing),
Eventlet, gevent
threads/single threaded
Result Stores
AMQP, Redis
memcached, MongoDB
SQLAlchemy, Django ORM
Apache Cassandra
Serialization
pickle, json, yaml, msgpack.
zlib, bzip2 compression.
Cryptographic message signing
![Page 12: Cooking pies with Celery](https://reader031.vdocuments.site/reader031/viewer/2022031922/5882bcf81a28abb2478b5279/html5/thumbnails/12.jpg)
Applicationfrom celery import Celeryfrom conf import Settingsfrom tasks.bake_pie import BakePie
APP_NAME = 'pie_fabric'
app = Celery(APP_NAME)app.config_from_object(Settings)app.tasks.register(BakePie())
![Page 13: Cooking pies with Celery](https://reader031.vdocuments.site/reader031/viewer/2022031922/5882bcf81a28abb2478b5279/html5/thumbnails/13.jpg)
Settingsclass Settings: BROKER_URL = 'amqp://guest:guest@localhost:5672//' CELERY_RESULT_BACKEND = 'redis://127.0.0.1:6379/0'
![Page 14: Cooking pies with Celery](https://reader031.vdocuments.site/reader031/viewer/2022031922/5882bcf81a28abb2478b5279/html5/thumbnails/14.jpg)
$ celery -A pie_fabric worker -l INFO
-------------- celery@amokrov v4.0.0rc2 (0today8)
---- **** -----
--- * *** * -- Linux-3.19.0-61-generic-x86_64-with-Ubuntu-14.04-trusty 2016-06-24 15:52:22
-- * - **** ---
- ** ---------- [config]
- ** ---------- .> app: pie_fabric:0x7efc8cd4c588
- ** ---------- .> transport: amqp://guest:**@localhost:5672//
- ** ---------- .> results: redis://127.0.0.1:6379/0
- *** --- * --- .> concurrency: 4 (prefork)
-- ******* ----
--- ***** ----- [queues]
-------------- .> celery exchange=celery(direct) key=celery
[tasks]
. bake_pie
![Page 15: Cooking pies with Celery](https://reader031.vdocuments.site/reader031/viewer/2022031922/5882bcf81a28abb2478b5279/html5/thumbnails/15.jpg)
Architecture
Client 1
Client 2
Worker 1Broker
Task Queue 1
Task Queue 2...
Worker 2
Result Backend
send tasks
send tasks
tasks
tasks
task result
task result
get task resultget task result
![Page 16: Cooking pies with Celery](https://reader031.vdocuments.site/reader031/viewer/2022031922/5882bcf81a28abb2478b5279/html5/thumbnails/16.jpg)
Фабрика пирожков
Интерфейс
Заказ
Осуществление заказа
Стряпание пирога
Выпекание пирога
Получение ингредиентов для теста
Ингредиенты Заготовка
Пирог
Создание теста
Получение начинки
![Page 17: Cooking pies with Celery](https://reader031.vdocuments.site/reader031/viewer/2022031922/5882bcf81a28abb2478b5279/html5/thumbnails/17.jpg)
Workflow
get flour
bake pie
get meat
seal pie
create dough
order pie
get milk
get aggs
![Page 18: Cooking pies with Celery](https://reader031.vdocuments.site/reader031/viewer/2022031922/5882bcf81a28abb2478b5279/html5/thumbnails/18.jpg)
The canvas
Wokrflow primitives
● group● chain● chord● map, starmap, chunks
![Page 19: Cooking pies with Celery](https://reader031.vdocuments.site/reader031/viewer/2022031922/5882bcf81a28abb2478b5279/html5/thumbnails/19.jpg)
Tasksfrom celery.app.task import Taskfrom celery.utils.log import get_task_logger
logger = get_task_logger(__name__)
class GetIngredient(Task):
name = 'get_ingredients'
def run(self, ingredient_name): logger.info('Getting {}'.format(ingredient_name))
l # код получения ингредиента
return ingredient
![Page 20: Cooking pies with Celery](https://reader031.vdocuments.site/reader031/viewer/2022031922/5882bcf81a28abb2478b5279/html5/thumbnails/20.jpg)
Workflowfrom celery.app.task import Taskfrom celery import canvas, signature
class OrderPie(Task): name = 'order_pie' def run(self): get_meat = signature('get_ingredients', args=('meat',)) ... dough_chord = canvas.chord([get_eggs, get_milk, get_flour], create_dough) components_group = canvas.group(dough_chord, get_meat)
return (components_group | seal_pie | bake_pie).delay()
![Page 21: Cooking pies with Celery](https://reader031.vdocuments.site/reader031/viewer/2022031922/5882bcf81a28abb2478b5279/html5/thumbnails/21.jpg)
Client side>>> from tasks.order_pie import OrderPie>>> r = OrderPie().delay()>>> r.id'58c5bb47-8fb3-4d4a-b2f5-8899520b5179'>>> r.ready()True>>> r.get()[['091f9ae7-561a-4781-a4d9-47bbcb121360', [['0865f66b-b89d-4ff3-a272-1f6b01d0a11f', None], None]], None]>>> r.backend.get_task_meta(r.get()[0][0]){'task_id': '091f9ae7-561a-4781-a4d9-47bbcb121360', 'children': [], 'traceback': None, 'result': 'baked pie with meat', 'status': 'SUCCESS'}
![Page 22: Cooking pies with Celery](https://reader031.vdocuments.site/reader031/viewer/2022031922/5882bcf81a28abb2478b5279/html5/thumbnails/22.jpg)
Routingfrom kombu import Exchange, Queue
class Router: def route_for_task(self, task, *args, **kwargs): route = {'exchange': Settings.EXCHANGE.name, 'routing_key': 'common'} if task in {'bake_pie', 'get_ingredients', 'seal_pie', 'order_pie'}: route['routing_key'] = 'fabric' elif task == 'create_dough': route = {'exchange': 'dough',
'routing_key': 'dough'} return route
![Page 23: Cooking pies with Celery](https://reader031.vdocuments.site/reader031/viewer/2022031922/5882bcf81a28abb2478b5279/html5/thumbnails/23.jpg)
Routingclass Settings: EXCHANGE = Exchange('pie_fabric, type='direct') CELERY_ROUTES = (Router(),) CELERY_QUEUES = ( Queue('pie_fabric.common', EXCHANGE, routing_key='common'), Queue('pie_fabric.fabric', EXCHANGE, routing_key='fabric'), )
--- ***** ----- [queues] -------------- .> pie_fabric.common exchange=pie_fabric(direct) key=common .> pie_fabric.fabric exchange=pie_fabric(direct) key=fabric
![Page 24: Cooking pies with Celery](https://reader031.vdocuments.site/reader031/viewer/2022031922/5882bcf81a28abb2478b5279/html5/thumbnails/24.jpg)
Second applicationclass Settings: BROKER_URL = 'amqp://guest:guest@localhost:5672//' CELERY_RESULT_BACKEND = 'redis://127.0.0.1:6379/0' EXCHANGE = Exchange('dough', type='direct') CELERY_QUEUES = ( Queue('dough', EXCHANGE, routing_key='dough'), )
app = Celery('dough')app.config_from_object(Settings)app.tasks.register(CreateDough())
$celery -A dough worker -P gevent --concurrency=1000 -l INFO
- ** ---------- [config]- ** ---------- .> app: dough:0x7f850ee22748- ** ---------- .> transport: amqp://guest:**@localhost:5672//- ** ---------- .> results: redis://127.0.0.1:6379/0- *** --- * --- .> concurrency: 1000 (gevent)-- ******* ---- --- ***** ----- [queues] -------------- .> dough exchange=dough(direct) key=dough
[tasks] . create_dough
![Page 25: Cooking pies with Celery](https://reader031.vdocuments.site/reader031/viewer/2022031922/5882bcf81a28abb2478b5279/html5/thumbnails/25.jpg)
Geventgevent is a coroutine -based Python networking library that uses greenlet to provide a high-level synchronous API on top of the libev event loop
gevent.monkey – Make the standard library cooperative
def _patch_gevent(): from gevent import monkey, signal as gsignal, version_info
monkey.patch_all()
![Page 26: Cooking pies with Celery](https://reader031.vdocuments.site/reader031/viewer/2022031922/5882bcf81a28abb2478b5279/html5/thumbnails/26.jpg)
Pollingdef run(self, *ingredients): logger.info('Ingredients: {}'.format(ingredients)) id = create_dough(ingredients) # Отправка запроса на приготовление теста while True: time.sleep(polling_timeout) if ready(id): # Проверяем готово ли тесто dough = get_dough(id) # Если готово, то забираем
return dough
![Page 27: Cooking pies with Celery](https://reader031.vdocuments.site/reader031/viewer/2022031922/5882bcf81a28abb2478b5279/html5/thumbnails/27.jpg)
Calling Tasks
applyExecute this task locally, by blocking until the task returns.
apply_async
Apply tasks asynchronously by sending a message.
delay
Shortcut to send a task message, but does not support execution options.
retry
Retry the task.
![Page 28: Cooking pies with Celery](https://reader031.vdocuments.site/reader031/viewer/2022031922/5882bcf81a28abb2478b5279/html5/thumbnails/28.jpg)
OptionsLinking (callbacks/errbacks)link
link_error
ETA and countdowncountdown
eta
Expirationexpires
Retry
retry=True,
retry_policy={
'max_retries': 3,
'interval_start': 0,
'interval_step': 0.2,
'interval_max': 0.2,
})
SerializersCompressionRouting options
![Page 29: Cooking pies with Celery](https://reader031.vdocuments.site/reader031/viewer/2022031922/5882bcf81a28abb2478b5279/html5/thumbnails/29.jpg)
Callbacksoptions = { 'link': app.signature('seal_pie'), 'link_error': app.signature('order_error')}for sub_task in options.values(): sub_task.set( **app.amqp.router.route(sub_task.options, sub_task.task, sub_task.args, sub_task.kwargs) )
![Page 30: Cooking pies with Celery](https://reader031.vdocuments.site/reader031/viewer/2022031922/5882bcf81a28abb2478b5279/html5/thumbnails/30.jpg)
Periodic Tasks
from celery.schedule import crontab
CELERYBEAT_SCHEDULE = { 'check-every-minute': { 'task': 'check_ingredients', 'schedule':crontab()}
$ celery -A pie_fabric beat
![Page 31: Cooking pies with Celery](https://reader031.vdocuments.site/reader031/viewer/2022031922/5882bcf81a28abb2478b5279/html5/thumbnails/31.jpg)
Concurrency
prefork (multiprocessing)
eventlet/gevent
threads/single threaded
![Page 32: Cooking pies with Celery](https://reader031.vdocuments.site/reader031/viewer/2022031922/5882bcf81a28abb2478b5279/html5/thumbnails/32.jpg)
Signals
Task Signals
App Signals
Worker Signals
Beat Signals
Eventlet Signals
Logging Signals
Command signals
Deprecated Signals
Worker Signalsceleryd_after_setupceleryd_initworker_initworker_readyworker_process_initworker_process_shutdownworker_shutdown
![Page 33: Cooking pies with Celery](https://reader031.vdocuments.site/reader031/viewer/2022031922/5882bcf81a28abb2478b5279/html5/thumbnails/33.jpg)
Ссылкиhttps://www.rabbitmq.com
http://docs.celeryproject.org
![Page 34: Cooking pies with Celery](https://reader031.vdocuments.site/reader031/viewer/2022031922/5882bcf81a28abb2478b5279/html5/thumbnails/34.jpg)
Cпасибо за внимание!
![Page 35: Cooking pies with Celery](https://reader031.vdocuments.site/reader031/viewer/2022031922/5882bcf81a28abb2478b5279/html5/thumbnails/35.jpg)
Вопросы?