celery vs mrq
TRANSCRIPT
![Page 1: Celery vs MRQ](https://reader033.vdocuments.site/reader033/viewer/2022051123/587c93841a28abfa5e8b59d5/html5/thumbnails/1.jpg)
Celery -> MRQ
LeadCooker2016
![Page 2: Celery vs MRQ](https://reader033.vdocuments.site/reader033/viewer/2022051123/587c93841a28abfa5e8b59d5/html5/thumbnails/2.jpg)
Task Queue
LeadCooker2016
![Page 3: Celery vs MRQ](https://reader033.vdocuments.site/reader033/viewer/2022051123/587c93841a28abfa5e8b59d5/html5/thumbnails/3.jpg)
Celery
LeadCooker2016
is good
![Page 4: Celery vs MRQ](https://reader033.vdocuments.site/reader033/viewer/2022051123/587c93841a28abfa5e8b59d5/html5/thumbnails/4.jpg)
Celery
LeadCooker2016
is bad
![Page 7: Celery vs MRQ](https://reader033.vdocuments.site/reader033/viewer/2022051123/587c93841a28abfa5e8b59d5/html5/thumbnails/7.jpg)
MRQ
LeadCooker2016
https://github.com/pricingassistant/mrq
GeventRedis
MongoDB
![Page 8: Celery vs MRQ](https://reader033.vdocuments.site/reader033/viewer/2022051123/587c93841a28abfa5e8b59d5/html5/thumbnails/8.jpg)
MRQ
LeadCooker2016
https://github.com/pricingassistant/mrq
• mrq-run • mrq-worker • mrq-dashboard
![Page 9: Celery vs MRQ](https://reader033.vdocuments.site/reader033/viewer/2022051123/587c93841a28abfa5e8b59d5/html5/thumbnails/9.jpg)
MRQ (task)
LeadCooker2016
https://github.com/pricingassistant/mrq
from mrq.task import Task import urllib2
class Fetch(Task):
def run(self, params):
with urllib2.urlopen(params["url"]) as f: t = f.read() return len(t)
![Page 10: Celery vs MRQ](https://reader033.vdocuments.site/reader033/viewer/2022051123/587c93841a28abfa5e8b59d5/html5/thumbnails/10.jpg)
MRQ (enqueue)
LeadCooker2016
https://github.com/pricingassistant/mrq
from mrq.context import set_current_config from mrq import config from mrq.job import queue_job
cfg = config.get_config( parser=None, config_type="run", sources=("file", "env") ) set_current_config(cfg)
queue_job('leadcooker.tasks.Fetch',[])
![Page 11: Celery vs MRQ](https://reader033.vdocuments.site/reader033/viewer/2022051123/587c93841a28abfa5e8b59d5/html5/thumbnails/11.jpg)
MRQ (schedule)
LeadCooker2016
https://github.com/pricingassistant/mrq
SCHEDULER_TASKS = [
# [every day]: Check trial user { "path": "leadcooker.tasks.CheckTrialUsersTask", "params": {}, "interval": 24 * 3600 },
# [every day]: Check expired campaigns { "path": "leadcooker.tasks.CheckExpiredCampaignsTask", "params": {}, "interval": 24 * 3600 },
# [every 6 hours]: Update mail answers/undelivers { "path": "leadcooker.tasks.UpdateAllMailStatusTask", "params": {}, "interval": 6 * 3600 } ]
![Page 12: Celery vs MRQ](https://reader033.vdocuments.site/reader033/viewer/2022051123/587c93841a28abfa5e8b59d5/html5/thumbnails/12.jpg)
MRQ
LeadCooker2016
https://github.com/pricingassistant/mrq
• Integrated dashboard • gevent worker • Job management • IO monitoring • Greenlet tracing • Builtin scheduler • Supervisord integration • Integrated memory leak debugger
Benefits
![Page 13: Celery vs MRQ](https://reader033.vdocuments.site/reader033/viewer/2022051123/587c93841a28abfa5e8b59d5/html5/thumbnails/13.jpg)
MRQ
LeadCooker2016
is not good enough
![Page 14: Celery vs MRQ](https://reader033.vdocuments.site/reader033/viewer/2022051123/587c93841a28abfa5e8b59d5/html5/thumbnails/14.jpg)
MRQ
LeadCooker2016
https://github.com/pricingassistant/mrq
RAW_QUEUES = { "leadcooker_timed_set": { "job_factory": lambda rawparam: { "path": "leadcooker.tasks.ProcessLeadTask", "params": { "timed_set": rawparam } } }, }
![Page 15: Celery vs MRQ](https://reader033.vdocuments.site/reader033/viewer/2022051123/587c93841a28abfa5e8b59d5/html5/thumbnails/15.jpg)
MRQ
LeadCooker2016
https://github.com/pricingassistant/mrq
/!\ MRQ is not yet ready for public use. Soon!
![Page 16: Celery vs MRQ](https://reader033.vdocuments.site/reader033/viewer/2022051123/587c93841a28abfa5e8b59d5/html5/thumbnails/16.jpg)
MRQ
LeadCooker2016
https://github.com/pricingassistant/mrq
RAW_QUEUES = { "leadcooker_timed_set": { "job_factory": lambda rawparam: { "path": "leadcooker.tasks.ProcessLeadTask", "params": { "timed_set": rawparam } } }, }
![Page 17: Celery vs MRQ](https://reader033.vdocuments.site/reader033/viewer/2022051123/587c93841a28abfa5e8b59d5/html5/thumbnails/17.jpg)
MRQ
LeadCooker2016
https://github.com/pricingassistant/mrq
from mrq.job import queue_raw_jobs import time
queue_raw_jobs("leadcooker_timed_set", { 'param': time.time()+3600, })
![Page 18: Celery vs MRQ](https://reader033.vdocuments.site/reader033/viewer/2022051123/587c93841a28abfa5e8b59d5/html5/thumbnails/18.jpg)
MRQ
LeadCooker2016
https://github.com/pricingassistant/mrq
from mrq.job import queue_raw_jobs import time
queue_raw_jobs("leadcooker_timed_set", { '{"lead_id": 234}': time.time()+3600, })
![Page 19: Celery vs MRQ](https://reader033.vdocuments.site/reader033/viewer/2022051123/587c93841a28abfa5e8b59d5/html5/thumbnails/19.jpg)
MRQ
LeadCooker2016
https://github.com/pricingassistant/mrq
RAW_QUEUES = { "leadcooker_timed_set": { "job_factory": lambda rawparam: { "path": "leadcooker.tasks.ProcessLeadTask", "params": { "timed_set": ujson.loads(rawparam) } } }, }
![Page 20: Celery vs MRQ](https://reader033.vdocuments.site/reader033/viewer/2022051123/587c93841a28abfa5e8b59d5/html5/thumbnails/20.jpg)
MRQ
LeadCooker2016
OR
• python-rq • rq-scheduler • rq-gevent-worker • rq-dashboard