叢林裡的小芹菜Celery in the Django

By Walter Liu

About me Learn Python since 2004 Was a Chief Technical Director in

Softstar. Now an Architect in Trend Micro Light Celery user for several months.

Celery is XDistributed

AsynchronousTask Queue

Have you met? Deliver thousands to millions e-mails to

your members? Many computation heavy queries at the

same time that blocks your web site or DB?

Query slow or too many connection to external API? (like Twitter/Facebook)

Long run tasks that occupied your web server resources?

Why Celery? POC without Celery

Add a “@celery.task” and the tasks become distributed tasks.

Multi-queue, routing, all configurable. Many utility functionality to help task

execution Group/Chord/Chain/Callback Retry/timeout Etc…..

from celery import Celery @celery.task( max_retries = 3, default_retry_delay=2 * 60, # retry in 2 minutes. rate_limit = 20, time_limit = 10, # hard time limit soft_time_limit = 5, ignore_result = True, acks_late = True, # make sure this task will be done. )def send_twitter_status(oauth, tweet): try: twitter = Twitter(oauth) twitter.update_status(tweet) except (Twitter.FailWhaleError, Twitter.LoginError), exc: raise send_twitter_status.retry(exc=exc)

Understand the personality of your celery What happen if the program throw an

exception? What happen if the program hard/soft

time limit? What happen if the hard/soft time limit

reached in your group/chord function? Many others.

Our pitfall/trouble cases Many, too many small tasks

For example 1 billion per day. Large task argument (workaround => store task at

cache) Send file directly

Better not wait another task in a task. Using Redis as AMQP and no HA. Hard time limit reached in Chord function. Multi-process and multi-thread problem in Windows Logging. Queue management

Some internet complain about Celery Only Python API change and break his program

2.5.x => 3.0.x Pylint

Some similar things Gearman Huey Advanced Python Scheduler

Recruiting – we’re using Front-end: not very limited. Backend

Python Django MongoDB Redis/Memcache Celery