distributed task processing with celery - pyzh
Post on 15-Apr-2017
97 Views
Preview:
TRANSCRIPT
Distributed Task Processing with Celery
Part II César Desales
Task granularity
¡ Example:
download and process data from 3 different sources.
¡ Parallellism vs Communication overhead
Resource leaks
¡ Why doesn't Python release the memory when I delete a large object? http://bit.ly/1yHUUKt
¡ –maxtasksperchild
¡ CELERYD_MAX_TASKS_PER_CHILD
Pooling external resources
¡ Connect to DB every time a task is executed?
¡ Pass non serializable objects to workers?
Limits
¡ Task dependencies -> Deadlocks
¡ Stale results
¡ Throttling
¡ Retries
¡ -time-limit or CELERYD_TASK_TIME_LIMIT
¡ –soft-time-limit or CELERYD_TASK_SOFT_TIME_LIMIT
Scheduling ¡ Cron syntax
¡ Celery beat: single worker doing the scheduling
Queues
¡ celery -A proj worker -Q reporting --concurrency=3
¡ celery -A proj worker -Q operations --concurrency=20
Autoscaling
¡ celery --app=myapp worker --autoscale=15,8
¡ How many workers?
¡ How log is a piece of string?
Monitoring ¡ celery -A proj status
¡ celery -A proj purge
¡ celery -A proj inspect stats
Monitoring (flower) ¡ pip install flower
¡ celery flower --broker=redis://foobar/0 --port=5555
¡ python -m webbrowser -t "http://localhost:5555/"
Results ¡ Avoid if possible
¡ CELERY_RESULT_BACKEND='redis://localhost:6379/0’
¡ CELERY_TASK_RESULT_EXPIRES=1800
¡ In synch and asynch mode
Thanks!
top related